feature/server-persistence #1
Loading…
Reference in a new issue
No description provided.
Delete branch "feature/server-persistence"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Three durability fixes on the durable server path: - Recover the in-memory log base from the oldest retained segment so a stream that trimmed across a segment boundary no longer restarts at base 0, which made fresh consumers pull empty gap records and underflowed the trim byte accounting. - Surface a panicking user codec from Stream.Publish as an error instead of a silent (PublishAck{}, nil) success; scoped to the encode call so the scratch buffer is returned to the pool exactly once. - Run the consumer watermark coalescer on its own fixed cadence, independent of the publish fsync policy, so watermarks reach disk even in the default per-publish-fsync mode and are not flushed only at shutdown. Adds regression tests for each: trimmed-restart base and a fresh consumer (store-level and full-server recovery), a panicking publish codec, and the cursor coalescer flushing on cadence without a close.High-signal review of feature/server-persistence (Phase 1-2 durability). The design is thoughtful and well-documented: stream-as-directory, derived base/next, double-buffered CRC watermark, and a thorough crash-ordering analysis. Recovery tests are strong. Two durability issues should be fixed before merge (a data-loss path under the documented crash model, and an exactly-once gap on the publish error path), plus lifecycle and minor concerns.
@ -249,0 +269,4 @@return all, nil}cursor = resp.streams[len(resp.streams)-1]If a server ever returns
hasMore: truewith an emptystreamspage (a buggy/foreign peer, or a limit underflow),resp.streams[len(resp.streams)-1]panics with index-out-of-range. It relies on a non-local server invariant. Guard the page before indexing: ifhasMore && len(resp.streams) == 0 { return nil, errors.New(...) }.@ -102,0 +138,4 @@}server.store = stst.start(ctx)IMPORTANT -
Serverhas noCloseand nothing shuts the store down:store.close()is only invoked directly in tests. In production the cursor coalescer and publishLoop goroutines run forever on NewServer's ctx and the open segment/consumer file handles are never closed or flushed, so the storage root cannot be cleanly reopened and fds leak across restarts. Add aServer.Close(or aWithStore-owned shutdown hook) that cancels the loops and flushes + closes every journal.@ -513,0 +625,4 @@// vanish or resurrect across a restart. Stream creation is rare, so holding// the server lock for the mount is acceptable.if s.store != nil {journal, err := s.store.createStream(req.name, req.maxAge)createStreammounts the stream directory (Mkdir + writeStreamMeta temp+rename+fsync + two consumer headers + syncStreamsDir) while holding the globals.lock. A slow or failed disk stalls every stream's create and all concurrent operations. The in-code comment acknowledges stream creation is rare, but consider performing the disk mount outside the server lock and reconciling the map under it, so a blocking fsync cannot serialize the whole server.@ -705,0 +892,4 @@// latency. The append's own interest/age trims delete segments via// trimAfterCursorAdvance and the trim hooks; nothing is written for a moved// base, since base/next are derived from the segment set.if err := persistPublish(stream, rec); err != nil {IMPORTANT -
appendDedupmutates the in-memory memlog and the dedup map beforepersistPublishruns. IfpersistPublishfails (fsync/I-O error), the request returns ErrInternal but the record is already in the log andstate.dedup[id] = rec.seqis set. A client retry with the same dedup ID then returnsduplicate: truewith the stored seq, so the client believes the record is durable when it never reached disk - after a restart the acked record is gone, breaking exactly-once storage. Fix: register the dedup mapping (and roll back the in-memory append if feasible) only after the durable write succeeds, so an error leaves the publish re-doable as a fresh append.@ -0,0 +987,4 @@// Delete oldest segments while at least two remain (never the newest) and// the oldest's end (its next segment's base) is at or below base.for len(j.segs) > 1 && j.segs[0].base+uint64(j.segs[0].count) <= base {BLOCKING -
trimToderives the oldest sealed segment's end fromj.segs[0].base + uint64(j.segs[0].count), but the sealedcountheader is written insealLockedvia WriteAt + Close with no fsync before the next segment and its directory entry are fsynced. This branch's own crash-ordering spec (plan.md) says the header count "cannot be the authority" for trim and that a sealed segment's end is the next segment's base. If a crash leaves a stale (understated) count on the oldest segment and the trim base later lands inside that segment's true record range,base+count <= basebecomes true and the segment is deleted even though records still inside the retained window would be dropped - silent loss of durable, acked records. Fix: derive the end from the next segment's base, e.g.end := j.segs[1].basewhenlen(j.segs) >= 2, deleting whileend <= base.Reviewing
ae2a198(head of feature/server-persistence). All five REQUEST_CHANGES comments from review #4 are addressed:trimTonow derives the oldest sealed segment's end from the NEXT segment's durable filename base (j.segs[1].base) instead ofj.segs[0].base + count. The sealedcountheader is an unsynced WriteAt+Close rewrite and can be stale after a crash, so it is no longer treated as the authority for trim. New regression testTestTrimRecoveryStaleCountKeepsRecordscorrupts seg0's count to 0 and confirms the retained records survive (fails on the old code).publishnow rolls back onpersistPublishfailure viastreamState.rollbackFailedPublish(drops the dedup mapping and, when the record is still the tail, removes it through the newmemlog.removeTail), so a retry of the same dedup ID is a fresh append rather than a misleadingduplicate:true. Covered byTestRollbackFailedPublishEnablesRedo.Server.Close()plus an auto-close watcher onloopCtx.Doneinstore.start;store.closeis idempotent (newclosedguard understreamMu), so the coalescer/publishLoop goroutines stop and journals flush/close cleanly.hasMore && len(resp.streams)==0before indexing the page, returningall,nilinstead of panicking.s.lock, with as.creatingreservation set so concurrent duplicate creates stay a no-op.Verified:
gofmt,go vet ./...,go build ./..., andgo test -count=1 ./...all pass. (Note: this Forgejo instance exposes no API/MCP action to setresolver=trueper thread — the resolve routes return 405 — so each thread should be marked resolved in the UI; code-wise the threads are all addressed by this commit.)AI code review — feature/server-persistence
Reviewed the durability implementation (store.go, state.go, server.go, memlog.go, protocol.go, conn.go and tests) against the golang skill passes. The design — stream-directory-as-storage-unit, base/next derived from the segment set, double-buffered CRC watermark,
*os.Rootconfinement — is coherent, well documented and the crash-ordering reasoning is mostly airtight. No vulnerabilities or data races found. Two real issues on the durable-write and recovery paths should be addressed before merge; one design nit.Important
seqbeforeappendRecoveredsizes its chunk array; a corrupt/tampered seq forces unbounded allocation during startup.Suggestion
3. Consumer create/delete do disk I/O (incl. fsyncs) under the server's global lock, contradicting the PR's own move of
createStream's disk mount outside that lock.@ -166,0 +197,4 @@// A never-acked consumer's only persistent state is watermark == 0, exactly// what re-creating it produces, so create is write-behind and self-healing.state.persistConsumerCreate(name)🟡 SUGGESTION — consumer create/delete do disk I/O while holding the server's global lock.
addConsumerIfRoomcallspersistConsumerCreate→consumerSlot(store.go:1415) anddeleteConsumercallspersistConsumerDelete→deleteConsumer, both of which do file writes and fsyncs — and these run while the caller (Server.createConsumer/deleteConsumer) holdss.lock. This reintroduces exactly what the PR fixed forcreateStream, whose directory mount is moved outside the server lock so "disk I/O does not block every stream operation" (createStream comment). Not a correctness bug, but a global-latency regression. Consider reserving the slot under the lock and doing the file write/fsync after releasing it.@ -293,0 +361,4 @@// publish already grew past it, only the dedup entry is removed; the leftover// non-durable record is retained in memory for the retained window rather than// corrupt the log.func (state *streamState) rollbackFailedPublish(rec record) {🟠 IMPORTANT — failed durable-publish rollback does not roll back the journal, desyncing the on-disk log from the in-memory log.
rollbackFailedPublishonly restores the memlog tail and the dedup entry; it never touchesstate.journal. ButappendPublish(store.go:841) has already written the frame to the open segment and bumpedopen.count(writeRecord, store.go:811) beforesyncLocked(store.go:941) is the step that can fail. On a write/fsync failure the segment object and file are left one record ahead of the memlog, which has been rolled back torec.seq.The retry (server.go:938) then re-publishes the same dedup ID as a fresh append, and the memlog reuses
rec.seq(its tail was removed), so a second frame with the same seq is appended to the segment. After a restart,loadRecordsreplays both frames intoappendRecovered, which overwrites the duplicate record and double-countsretainedBytes— corrupting byte-budget accounting and violating the "sequence number never reused" invariant the recovery logic depends on.Condition: any failed durable write/sync — precisely the failure this rollback exists to handle.
TestRollbackFailedPublishEnablesRedobuilds a nil-journal stream, so it never exercises the desync.Fix: on failure also rewind the journal segment (reset
open.count, truncate the file back to the pre-append offset, cleardirty), and add a test with a real*os.Root/segment; or stop serving the stream once a publish cannot be made durable.@ -0,0 +768,4 @@return nil, err}log.appendRecovered(rec)🟠 IMPORTANT — recovery does not validate a frame's
seqbefore feeding it toappendRecovered, so a corrupt/tampered segment can OOM the server during startup.appendRecoveredsizes its chunk array to the slotrec.seqlands on (for len(chunks) <= pos/size { append }) and its doc comment explicitly says the caller must validaterec.seqfirst ("an out-of-range value would size the chunk array to match"). This caller,loadRecords, never boundsrec.seq. A corrupt frame withseqfar abovenextallocates a huge chunk array; one withseqbelowbasemakesposition()(seq - base + base%size) underflow uint64 to a huge index, also forcing a massive allocation.That contradicts the documented contract that a corrupt frame "refuses that stream rather than misreading it" — only a decode error refuses; a well-formed frame with a bad seq is accepted.
Fix: before
appendRecovered, reject anyrec.seqoutside the segment's expected window (bound it) and return a corruption error.Reviewing
d609bc4(head of feature/server-persistence). The three REQUEST_CHANGES comments from review #6 are addressed:appendPublishnow captures the pre-append file offset/count/lastTS and, on a failed write or durable sync, callsrewindAppendLockedto truncate the open segment back to exactly its prior state. A failed publish no longer leaves the on-disk log a frame ahead of the in-memory log, so a retry reuses the same sequence number as a fresh append and a restart cannot replay two frames with the same seq (bytes double-counted / record overwritten). A test-onlyfailSyncseam insyncLockedreproduces the durable-sync failure on a real*os.Rootsegment; newTestPublishSyncFailureRewindsJournalChainconfirms the rewind and that recovery yields exactly one seq-3 frame.loadRecordsnow validates each frame'sseqagainst the segment's[base, base+segmentRecords)window beforeappendRecovered, returning a corruption error for an out-of-window seq instead of forcing an unbounded chunk allocation at startup. NewTestRecoveryRejectsOutOfWindowSeqcovers it (seq chosen to pass the segment-level count scan so only the per-frame validation catches it).createConsumer/deleteConsumernow reserve the slot under the server's global lock but run the durable slot write, tombstone, and watermark fsyncs after releasing it, matching the PR's own move ofcreateStream's disk mount.addConsumerIfRoomreports whether the consumer was created;deleteConsumerreports existed/moved/base for the caller to persist.consumerStore.consumerSlot/deleteConsumerare now nil-safe against the concurrentdeleteStream-close this allows.Verified:
gofmt,go vet ./...,go build ./..., andgo test -count=1 -timeout 120s .all pass (the full package suite completes in ~1s; the env QUIC/libp2p hang is unrelated). (Same resolver caveat as before: this Forgejo build exposes no API/MCP action to setresolver=trueper thread, so mark the threads resolved in the UI; code-wise all are addressed.)Root-caused and fixed the flaky
TestSingleConsumeLoopOverTheWirehang.Root cause (real, not environmental): when a consume loop exits it sends an explicit
releaseLease. But its final in-flight pull (already written to the server) can be processed after the release lands.acquireLeasesaw the freshly-zeroed lease and re-acquired the same token, orphaning the lease. The replacement loop's pull then gotErrConsumerBusy, which is terminal in the loop, so it died and the test hung waiting for takeover.Fix (
state.go): track aretiredLeasetoken. Once a loop releases its lease, that token can never be acquired again; a stale pull carrying it is refused as busy. A genuine replacement loop (fresh token) still acquires normally.Also hardened the test (
integration_test.go): capture the first loop's own error channel before the second loop replaces it (so the wait is on the first loop's stop/release, not the already-closed second channel), and surface a replacement-loop terminal error instead of hanging.Verification:
go build,go vet,gofmtclean. Full suite passed 30 consecutive runs at-parallel 4, 15 at-parallel 8, plus 20 solo runs of the target test. The prior flake (previously dismissed as transport/libp2p noise) no longer reproduces.@ -102,0 +288,4 @@// Nothing is recovered at construction: a stream opens on first access.if server.storeRoot != nil {st := newStore(server.storeRoot, server.logger, server.flushInterval)server.store = stIMPORTANT - nil deref on shutdown. In acquire, a first-touch inserts a residentStream with stream == nil into s.streams before store.openStreamState completes. If Server.Close() runs in that window, rs.stream is nil and stream.stopAgeTrim() panics (nil receiver). Fix: guard the loop (skip nil streams); the still-loading journal is covered by store.close/flush.
@ -64,0 +185,4 @@ch := rs.loadings.lock.Unlock()select {SUGGESTION - evictIdleStreams has no runtime caller: only tests invoke it directly. So the eviction half of lazy residency is inert in production - streams load on first touch but nothing ever evicts them, leaving idleEvictable/residentStream.evict dead at runtime and resident memory unbounded on a long-running server. If auto-eviction is intended (Phase 4/5), wire a scheduler; otherwise document that eviction is test-only for now.
@ -228,3 +500,4 @@target = rs.stream}}BLOCKING - deleteStream builds a filesystem path from an unvalidated req.name, unlike createStream (line 431) and createConsumer (line 665) which call validName. On a durable server a non-resident (cold) stream falls through to s.store.removeStream(req.name) -> removeQuiet(streamsDir + '/' + name) -> root.RemoveAll('streams/'+name). Go's os.Root.splitPath strips empty and '.' path components, so a frame carrying name='.' (or an empty name) resolves to the streams/ directory itself and removes every stream - a single unauthenticated delete frame destroys all durable data. Fix: reject with validName(req.name) (errInvalidFrame) at the top of deleteStream (and make getStream/streamExists reject too), mirroring createStream.
@ -0,0 +416,4 @@}log, err := j.loadRecords(ctx)if err != nil {IMPORTANT - journal/FD leak on recovery failure. openStream registers the journal in store.streams and leaves the consumer slot files + open segment file open; openStreamState then returns an error from readStreamMeta/loadRecords without closing it. On the caller side, Server.acquire only deletes the server-level streams entry on error - it never closes the journal or removes it from store.streams. So a persistently corrupt/torn stream leaks one journal + open FDs into the store live set on every touch, and flushConsumers/flushJournals keep syncing it. Fix: deregister and j.close() on the post-registration error path (or register the journal only after the full open succeeds).
AI code review of feature/server-persistence at head
433d315(Phase 3 lazy residency on top of the already-reviewed Phases 1-2). Reviewed the full current diff (store.go, state.go, server.go, memlog.go, protocol.go, conn.go, stream.go + tests) against the golang review passes. The design is coherent and well-documented: stream-directory-as-storage-unit, derived base/next, double-buffered CRC watermark, *os.Root confinement, single-flight lazy acquire/release, and strong recovery tests. Prior review threads (publish-journal desync, recovery seq validation, consumer I/O under global lock, trimTo stale-count, lifecycle) all appear addressed in the current head.New findings in the Phase 3 lazy-residency surface:
No go.mod changes (no new dependencies; no CVE surface added).
@ -102,0 +288,4 @@// Nothing is recovered at construction: a stream opens on first access.if server.storeRoot != nil {st := newStore(server.storeRoot, server.logger, server.flushInterval)server.store = stIMPORTANT - nil deref on shutdown. In acquire, a first-touch inserts a residentStream with stream == nil into s.streams before store.openStreamState completes. If Server.Close() runs in that window, rs.stream is nil and stream.stopAgeTrim() panics (nil receiver). Fix: guard the loop (skip nil streams); the still-loading journal is covered by store.close/flush.
@ -64,0 +185,4 @@ch := rs.loadings.lock.Unlock()select {SUGGESTION - evictIdleStreams has no runtime caller: only tests invoke it directly. So the eviction half of lazy residency is inert in production - streams load on first touch but nothing ever evicts them, leaving idleEvictable/residentStream.evict dead at runtime and resident memory unbounded on a long-running server. If auto-eviction is intended (Phase 4/5), wire a scheduler; otherwise document that eviction is test-only for now.
@ -228,3 +500,4 @@target = rs.stream}}BLOCKING - deleteStream builds a filesystem path from an unvalidated req.name, unlike createStream (line 431) and createConsumer (line 665) which call validName. On a durable server a non-resident (cold) stream falls through to s.store.removeStream(req.name) -> removeQuiet(streamsDir + '/' + name) -> root.RemoveAll('streams/'+name). Go's os.Root.splitPath strips empty and '.' path components, so a frame carrying name='.' (or an empty name) resolves to the streams/ directory itself and removes every stream - a single unauthenticated delete frame destroys all durable data. Fix: reject with validName(req.name) (errInvalidFrame) at the top of deleteStream (and make getStream/streamExists reject too), mirroring createStream.
@ -0,0 +416,4 @@}log, err := j.loadRecords(ctx)if err != nil {IMPORTANT - journal/FD leak on recovery failure. openStream registers the journal in store.streams and leaves the consumer slot files + open segment file open; openStreamState then returns an error from readStreamMeta/loadRecords without closing it. On the caller side, Server.acquire only deletes the server-level streams entry on error - it never closes the journal or removes it from store.streams. So a persistently corrupt/torn stream leaks one journal + open FDs into the store live set on every touch, and flushConsumers/flushJournals keep syncing it. Fix: deregister and j.close() on the post-registration error path (or register the journal only after the full open succeeds).
All four findings from the Phase 3 review (head
433d315) are addressed inf7c2dca.deleteStreamandgetStreamnow reject any name failingvalidNamewitherrInvalidFrame, mirroringcreateStream. A frame naming'.'/empty can no longer reachstore.removeStream/RemoveAlland wipe the storage tree. Covered byTestDeleteStreamRejectsInvalidName(deletes'. ``, empty,..,a/b,.hidden` are refused; the valid stream survives).Server.Closenow skips resident streams whose load is still in flight (stream == nil); the still-loading journal is covered bystore.close. Covered byTestServerCloseSkipsLoadingStream.openStreamcloses the consumer slot files ifloadLog/consumers.loadfails, andopenStreamStatederegisters (journal.evict) + closes the journal when a post-registration read (readStreamMeta/loadRecords) fails. A corrupt stream no longer leaks a journal + open FDs into the store live set on every touch. Covered byTestOpenStreamFailureDoesNotLeakJournal(corrupts stream.meta, asserts the journal is not left registered).residencyLoopgoroutine launched inNewServer(durable servers only) that periodically evicts resident streams idle beyondstreamIdleEvictThreshold(2m), so a long-running server reclaims cold streams. Phase 4 will replace the fixed idle timeout with the node storage budget as the eviction driver.Verified:
gofmt,go vet ./...,go build ./..., and the full package suite (5 runs at-parallel 8) all pass.Reviewed the full diff (store.go, server.go, state.go, memlog.go, conn.go, protocol.go, and the new test files) against the golang-* skills (safety, concurrency, error handling, correctness). The durable store is generally well built: careful validation of every on-disk header, double-buffered CRC watermarks, single-flight lazy residency, rollback on failed durable writes, and strong coverage (recovery, lazy residency, store tests).
Two issues to address before merge — a data-integrity bug in the durable publish path (#1) and a budget gap for durable + lazy-residency deployments (#2) — plus one minor suggestion. The rest of the changes look correct. Not approving yet; fix #1 first.
@ -64,0 +147,4 @@s.streams[name] = rss.lock.Unlock()stream, err := s.store.openStreamState(ctx, name)🟡 SUGGESTION — a request racing a durable
createStreamcan get a spurious error.acquire(first-touch) does not consults.creating, so a publish/get/pull for a name whose directory mount is still in flight callsopenStreamStatewhile the directory orstream.metamay not exist yet, returningErrStreamNotFound(or failing mid-creation). Transient, not corrupting, but a client that issues a publish concurrently with a create on another connection can observe a spurious error. Consider returning a get-or-create-style no-op / retry while the name is ins.creating.@ -191,10 +500,10 @@ func (s *Server) enforceStorageBudget() {func (s *Server) totalRetainedBytesLocked() uint64 {var total uint64🟠 IMPORTANT — on a durable server the storage budget is only enforced against resident streams.
totalRetainedBytesLockedsums onlys.streams(resident), and this PR evicts cold streams to disk (evictIdleStreams), so a durable server's idle streams keep bytes on disk without counting against thedefaultStorageBytes(256 MiB) bound.enforceStorageBudget's re-scan also reads the resident total, so it sees ≤ budget and never evicts. Net effect: the advertised 256 MiB default is not a real bound once streams are lazy-evicted. This looks like the documented Phase-4 "global storage budget" disk sweep, but it's worth confirming it is consciously deferred in this PR — a deployment relying on the default bound on a durable server won't actually get it.@ -687,0 +1150,4 @@// publish is on disk. The write happens outside the stream lock; the// per-stream journal serializes publishes to the same stream on disk// latency. The append's own interest/age trims delete segments via// trimAfterCursorAdvance and the trim hooks; nothing is written for a moved🔴 BLOCKING — concurrent publishes to the same stream can write the journal out of order and corrupt (or refuse) the stream on restart. Sequence numbers are assigned under
stream.lockinsideappendDedup(server.go ~1141), but the durable write happens later under the journal's ownjournal.mu(persistPublish→appendPublish, store.go ~921). Two different mutexes, so the order in which they are acquired is not tied to the order sequence numbers were allocated.Concretely: two connections publish to "orders" concurrently. G1 is assigned seq N, G2 seq N+1 (in-memory order N, N+1). But G2 can take
journal.mufirst and persist N+1, then G1 persists N — the segment file holds [N+1, N].appendPublishwrites whateverrec.seqit is given, androllLockedrolls a new segment atrec.seq, with no check thatrec.seq == journal.next().• If the stream was empty (open==nil), G2 rolls a segment at base N+1, then G1 writes a frame with seq N into it. On restart
loadRecordsrejectsrec.seq < seg.baseand the whole stream refuses to load (unreadable).• Otherwise the segment ends with non-monotonic frames [N+1, N]; on recovery
appendRecoveredreads N+1 (next=N+2) then N (next regresses to N+1), leaving a record at/abovenextand a wrongnextafter restart.Fix: serialize the in-memory append and the durable append under one per-stream writer so on-disk order always equals allocation order — e.g. hold a per-stream mutex across
appendDedup+persistPublishfor durable streams, or haveappendPublishverifyrec.seq == j.next()and roll/refuse otherwise. The existing rollback path handles a failed write but not an out-of-order one.All three review-#10 comments are addressed in the new head
3064914(on top of the Phase 4 storage-budget commit). Please mark the threads resolved in the UI (this Forgejo build exposes no resolver API).#105 (BLOCKING — concurrent publish journal ordering). The durable publish path now holds the journal's per-stream writer lock (
journal.mu) across both the in-memory append (appendDedup, which allocates the seq understream.lock) and the durable write, so the order records reach disk always equals the order their seqs were allocated. The old code took two unrelated mutexes and could let a later-allocated seq reach disk first.appendPublishis split intoappendPublishLockedfor this path. Regression testTestConcurrentDurablePublishOnDiskOrderruns 16 goroutines × 100 publishes and asserts strict seq order plus correctnextafter a restart.#106 (IMPORTANT — budget only against resident streams). Addressed by the preceding Phase 4 commit (
72b8c51):WithStoragenow enforces the node total via a disk sweep over cold, non-resident streams (store.budgetSnapshot), and evicts the globally-oldest sealed segments, so the 256 MiB default is a real bound under lazy residency. Covered bynode_budget_test.go.#107 (SUGGESTION — acquire races a durable create).
createStreamnow reserves the name as a single-flight loading residentStream (the same markeracquirealready waits on) instead of a separates.creatingmap, so a publish/get/pull racing a durable create waits for the mount instead of erroringErrStreamNotFoundmid-creation. Thecreatingfield is dropped. Covered byTestAcquireWaitsForStreamBeingCreated.Verified:
gofmt,go vet ./...,go build ./..., and the full suite (go test -count=1 ./...) pass; the new concurrency test is stable across-count=3.Code review — line-by-line, with reproduction
Reviewed pull request #1 of
norobo/webstream(feature/server-persistence, head3064914, basemain894f169; +6681/−545 across 24 files) againstplan.md— the design document for this "server persistence" work that lives at the repository root and specifies the on-disk layout (one directory per stream understreams/), the durability table, lazy residency, and the two consumer slot files.Verification performed:
gofmt -lclean,go vet ./...clean,go test -count=1 ./...passes. Findings marked [reproduced] were confirmed with throwaway Go tests added to the package and then removed (working tree clean); each reproduction is described inline so it can be re-added as a regression test.Terms used below. A watermark (
consumerState.processedBefore) is the durable, exclusive record sequence number up to which a consumer has confirmed processing. A segment is one file of a stream's record log, named by the sequence number of its first record (20-digit zero padded). A slot is a fixed-stride entry inconsumers.meta(the consumer-name → slot index) or incursors.meta(the CRC-protected watermark values). "Resident" means materialized in the server's in-memory stream map.BLOCKING
B1.
consumerStore.deleteConsumercan free a slot a concurrent create just took — durable consumer entry and watermark loststore.go:1948The function mutates the free list and zeroes the cursor slot under
cs.mu, then releases the mutex in order to fsynccursors.meta, then re-acquires it and writes thefreestate byte atidxOff. In the gap a concurrentconsumerSlot(store.go:1818) pops that same slot fromfreeSlots, writes its own entry at the sameidxOff, and setsliveSlots[slot] = true— after which the delete's re-acquired lock writesconsumerFreeover the new entry.[reproduced] A 64-pair concurrent
consumerSlot/deleteConsumerprobe (all names distinct) left 39 of 64 newly created consumers live in RAM (nameToSlot,liveSlots) but absent fromconsumers.metaafter a reload. The window is an fsync wide, so this is easy to hit under load.The impact is exactly the failure the design forbids: on restart that consumer vanishes, the client re-creates it at watermark 0, and every record in
[base, W)is reprocessed — losing the one value the store exists to protect (plan.md: "losing a watermark would duplicate processing"). The reverse interleaving also exists: the delete'sliveSlots[slot] = falsecan land after the create's.Fix: keep the free-list mutation, both
WriteAtcalls and their ordering inside one lock hold, and move only theSynccalls outside it — or re-verify!cs.liveSlots[slot](and that no name maps toslot) before writingfree, skipping the write when the slot was reclaimed.Regression test: concurrent create/delete, then reopen the store and assert the RAM live set equals the reloaded one.
B2. Unvalidated
req.streamreaches the filesystemserver.go:135(acquire),store.go:368(openStream)createStream(server.go:884),deleteStream(server.go:983) andgetStream(server.go:1088) callvalidName;publish,pull,ack,createConsumer,deleteConsumer,getConsumerandreleaseConsumerLeasedo not. They passreq.streamstraight toacquire→store.openStreamState→streamsDir + "/" + namewithO_CREATEonconsumers.meta/cursors.meta.[reproduced] Against a durable store:
*os.Rootprevents escaping the storage root (so this is not a sandbox break, and only names whose parent already exists resolve at all), but a wire-supplied field creates files instreams/and in the storage root, where only stream directories belong. Fix at the root cause: validate once inacquire, which every request path funnels through.B3.
listStorePagetreatslimit == 0aslimit == 1server.go:1070listStreamsRequest.limitis documented as "zero means no cap"; withlimit == 0the+1probe asks for exactly one name, truncates to one, and reportshasMore == false— an uncapped request silently returns a single name from a durable server. The in-memory path (listStreamNames,server.go:1040) handles 0 correctly, so the two backends disagree on the same wire message.IMPORTANT
I1. Cold-stream delete is not tombstoned
store.go:459(store.removeStream)journal.remove(store.go:1170) performs the documented rename to.<name>+syncStreamsDir+ best-effortRemoveAll.removeStream, used bydeleteStreamwhen the stream is not resident (server.go:1000-1005), does a bareRemoveAll+ sync. A crash mid-RemoveAllleaves a half-deleted live directory:listStreams/getStreamreport the stream as existing (it is a valid-named directory) whileopenStreamStatefails on the missingstream.meta. Bothplan.md("Delete: rename the stream directory to a tombstone name…") and the README claim the tombstone path. Share one helper between the two.I2.
journal.closedis written and never read; a publish racing delete recreates the stream directorystore.go:162(declaration),store.go:1181(assignment),store.go:937(appendPublishLocked)removenilsj.openand setsclosedunderj.mu, then releasesj.mubefore the rename andRemoveAll. A publish that already holds a stream handle (refs > 0, so eviction is excluded by design) can takej.muafter that, hitj.open == nil, and enterrollLocked(store.go:1003) →MkdirAll("streams/<name>/log")+ create a segment — recreatingstreams/<name>/after the rename, racing theRemoveAllof the tombstone. The result is a phantom stream that lists and stats as existing but fails to open.consumerSlot/deleteConsumeralready guard this withidxFile == nil; the segment path has no equivalent guard, andclosedis the field that was clearly intended to be it. Return an error fromappendPublishLocked/flushJournalwhenj.closed, and keep the rename plusRemoveAllinsidej.mu.I3. Budget enforcement does whole-node disk I/O and per-segment fsyncs under the global server write lock
server.go:514(enforceDurableStorageBudget),server.go:546(evictDurableToBudgetLocked),store.go:1350(budgetSnapshot)budgetSnapshotreaddir + stat + header-reads every stream directory and segment, and it is called whiles.lockis held (server.go:519for the seed,server.go:527for enforcement).deleteSegment(store.go:1545), which fsyncsstreams/<name>/log/; each resident one callsreleaseBudgetTrim→persistWatermark→trimTo→syncLogDir, another fsync — still unders.lock.One publish that trips the budget therefore stalls every other stream's publish/pull/ack for the duration of N fsyncs plus a full-directory sweep. The snapshot needs no lock (it re-reads disk) and the deletions only need the per-stream
journal.mu; take the write lock for the resident reconciliation only, and bound the batch.I4. The min-cursor cache is gone for durable streams
store.go:1917(minLiveWatermark), reached fromstate.minConsumerCursorLockedThe
journal != nilearly return makes every trim — that is, every publish and every ack — takecs.muand iterate the wholenameToSlotmap, while the in-memory path keepsminCursor/minCursorValidprecisely to avoid this. WithmaxConsumersPerStreamraised to 65535, one record now costs up to 65k map iterations plus a mutex that ack also contends on. The invalidation rule already written for the cached path (watermarks only rise; create/delete invalidate) applies unchanged to the resident watermark array.I5. A segment header's
lastTSis never finalized — the budget's global order key is wrongstore.go:1059(sealLocked),store.go:1003-1046(rollLocked),store.go:1533(segmentLastTS)rollLockedwrites the header withfirstTS == lastTS == rec.timestamp, the timestamp of the segment's first record.writeRecord(store.go:907) updateslastTSin memory only, andsealLockedrewrites only the count at offset 16. So for every sealed segment, offset 28 holds a first-record timestamp, andbudgetSnapshotsorts its "globally oldest segments" candidates by that value. A segment that ends newer than another stream's segment can therefore be evicted first, diverging from the DiscardOld order the plan specifies.loadSealedSegmentrecovers the same wrong value. Fix: writelastTSat offset 28 insealLocked, which already rewrites the header; add a test asserting a sealed header's count and timestamps.MINOR / CODE QUALITY
streamExistscomment is repeated atstore.go:1565glued onto theconsumerStoredoc block (// streamExists reports whether… consumers.meta (the…type consumerStore struct). Looks like a botched merge. Related:journal.next()(store.go:916) exists only forstore_test.go:276— production API kept alive by a test.consumerStore.loadvalidation gaps (store.go:1673):nameLen < 0is dead code (the value comes from auint8);nameLen == 0yields a live, reachable consumer literally named""; cursor-slot uniqueness is not checked, so two live index entries can claim the same slot andliveSlots[slot]then governs the wrong consumer's watermark. The sanitizingWriteAtof non-live slots is also not fsynced and leaveswatermarks[i]/slotStates[i]at zero without marking the copy state, so "zeroed before the slot is handed out again" rests on writeback luck rather than the stated invariant.scanSegmentallocates an unvalidated length (store.go:699):make([]byte, length)from the 4-byte prefix before bounding it against the remaining file size (whichloadRecordsdoes do at store.go:840). A corrupt tail of garbage rather than zeros allocates up to 4 GiB. The guard is alsolength < wireByteSize, andwireByteSize == 1(protocol.go:32) while a record's minimum encoded size is 24 bytes, so a truncated frame is refused as corruption instead of being truncated as a torn tail.maxStreamsPerServercounts only resident streams (server.go:281, checked atserver.go:908). Under lazy residency with idle eviction, a durable node's on-disk stream count is effectively unbounded. The compensating control the plan promises — "a per-stream reserve counts toward the budget" — does not exist in the code:sweepStream(store.go:1435) sums real file sizes only (48 bytes of headers per stream), so the 256 MiB default admits millions of empty streams.Server.Closewritess.store = nilunguarded (server.go:339) whileacquirereads that field (server.go:135), a data race once a handler is in flight (the doc says the server must not be used afterwards, but-racewill flag it). Alsostore.openStreamdoes not checkstore.closed, so a load starting duringcloseregisters a journal after the close loop's snapshot and is never flushed or closed.listResponseOverhead = 9(protocol.go:54) versus an actual fixed size of 5 (auint32count plus thehasMorebyte). Conservative, but the comment claims exactness.listStreamsreads and sorts the whole directory per page (store.go:1251):limitis applied afterReadDir(-1)andslices.Sort, so paging an unbounded set is O(n log n) per page.Reviewed and found correct
server.lock → state.lockandjournal.mu → state.lock; everypersistWatermark*/trimTocall happens after releasingstate.lock, and the publish error path unlocksjournal.mubeforerollbackFailedPublish. No inversion, no deadlock.journal.muacrossappendDedupplusappendPublishLockedis the right shape, andrewindAppendLockedcorrectly restores offset, count andlastTSso a retry reuses the same sequence number. The roll itself is not undone, but stays consistent because the new segment's base is derived fromrec.seq.position/trimBeforebookkeeping holds for a non-chunk-aligned segment base (base 300 ⇒ offset 44);removeTail's tail check is exact; the new chunk-array bounds guards are necessary.decodeDoubleBuffered(higher valid copy wins, both torn ⇒ 0 ⇒ redelivery) andencodeStaleCopy's alternating copy are correct; a torn write cannot read high within a slot.advanceIfkeeps cursor and watermark in one critical section, and theMaxUint64ack guard prevents cursor wrap.resp.cursor, addinghasMore) is consistent across encode, decode and the fuzz seeds;maxRecordSize/listResponseOverheadbound the frames.Suggested regression tests: concurrent create/delete-consumer asserting RAM == disk live sets (B1);
limit == 0for both backends (B3); sealed-segment header assertions for count andlastTS(I5); delete versus concurrent publish asserting no stream directory is recreated (I2).@ -64,0 +132,4 @@// caller must call release(name) when done. The stream is not evictable while a// handle is held, so in-flight code can safely use the returned *streamState// across the acquire boundary.func (s *Server) acquire(ctx context.Context, name string) (*streamState, error) {BLOCKING —
acquireis the funnel for every request path that carries a stream name, yet it never callsvalidName(createStream,deleteStreamandgetStreamdo).req.streamof.or..therefore reachesopenStreamState→streamsDir + "/" + namewithO_CREATE, which I reproduced creatingconsumers.meta/cursors.metainstreams/and in the storage root.*os.Rootconfines it, so it is not a sandbox escape, but a wire field must not create files outside a stream directory. Validate here, once.@ -189,0 +543,4 @@// directly. A segment whose stream is mid-load is left for a later sweep, since// the in-flight load would otherwise race the deletion. The caller holds// s.lock.func (s *Server) evictDurableToBudgetLocked(total uint64, deletable []budgetSegment) {IMPORTANT — called with
s.lockheld, this performs per-segment fsyncs (deleteSegment, andreleaseBudgetTrim→persistWatermark→trimTo→syncLogDir) and, via the caller, a full-nodebudgetSnapshotreaddir/stat sweep. A single publish that trips the budget stalls every other stream's publish/pull/ack for N fsyncs plus that sweep. The snapshot needs no lock; the deletions need only the per-streamjournal.mu.@ -557,0 +1067,4 @@// listStorePage lists a page of stream names from the storage directory,// fetching limit+1 to detect the page boundary.func (s *Server) listStorePage(req listStreamsRequest) ([]string, bool, error) {BLOCKING —
int(req.limit)+1turns the documented "zero means no cap" into a one-name page: withlimit == 0this askslistStreamsfor 1 name, truncates to it, and reportshasMore == false. The in-memory path (listStreamNames) handles 0 correctly, so the two backends disagree on the same request.@ -0,0 +456,4 @@// removeStream deletes a stream directory for a stream that is not resident, so// a delete of a cold stream does not resurrect on the next load.func (s *store) removeStream(name string) {IMPORTANT — unlike
journal.remove(store.go:1170), this cold-stream delete skips the tombstone rename: a crash mid-RemoveAllleaves a half-deleted directory thatlistStreams/getStreamreport as existing (valid name, is a directory) whileopenStreamStatefails on the missingstream.meta.plan.mdand the README both promise the tombstone path; share one helper.@ -0,0 +1056,4 @@// file, turning it into a sealed (non-newest) segment. The count is advisory; a// sealed segment's end is derived from the next segment's base. The caller holds// journal.mu.func (j *journal) sealLocked() error {IMPORTANT —
sealLockedfinalizes only the count (offset 16). The header'slastTS(offset 28) is written once inrollLockedwith the first record's timestamp and never updated, yetsegmentLastTS(store.go:1533) feeds it tobudgetSnapshotas the globally-oldest ordering key, so eviction order is wrong across streams. WritelastTShere too and assert it in a test.@ -0,0 +1167,4 @@// remove deletes the stream directory (tombstone + RemoveAll) so a deleted// stream does not resurrect, and drops it from the store's live set.func (j *journal) remove() {IMPORTANT —
closedis set here and read nowhere, and the rename/RemoveAllrun afterj.muis released. A publish that already holds a stream handle (refs > 0, so eviction is excluded by design) can then enterrollLocked→MkdirAll("streams/<name>/log")and recreate the directory after the rename, producing a phantom stream that lists as existing but fails to open. Guard the segment write path onj.closedasconsumerSlot/deleteConsumerguard onidxFile == nil.@ -0,0 +1945,4 @@// and fsyncs cursors.meta, then marks the consumers.meta entry free and fsyncs// it, before the ack returns. A crash between leaves a live consumer with a// zeroed watermark (safe redelivery), never a freed slot carrying a watermark.func (cs *consumerStore) deleteConsumer(name string) {BLOCKING — the free-list mutation and the
freestate byte are written under two separate lock holds, with an fsync in between. A concurrentconsumerSlotcan pop the slot and write its own entry at the sameidxOffin that gap, and this function then overwrites it withconsumerFree. Reproduced: 39 of 64 concurrent create/delete pairs left a consumer live in RAM but absent fromconsumers.metaafter reload — its watermark is lost, so the client restarts at 0 and reprocesses[base, W). Keep both writes and the free-list mutation in one critical section (move only theSyncs out), or re-check!cs.liveSlots[slot]before writingfree.@ -0,0 +1914,4 @@// zero when none is live. It is the durable source of interest trim, which must// see every consumer (including ones never materialized as a consumerState): a// never-materialized consumer still holds records a trim must not drop.func (cs *consumerStore) minLiveWatermark() uint64 {IMPORTANT — this is reached from
streamState.minConsumerCursorLockedon every trim, i.e. every publish and every ack, takingcs.muand iterating the wholenameToSlotmap. The in-memory path caches the minimum (minCursor/minCursorValid) for exactly this reason; at the new 65535-consumer cap a single record can cost 65k iterations plus lock contention against acks.Review #20 resolved — commit
c4b69a0All eight inline comments plus the MINOR items from the review description are addressed on the updated head (
c4b69a0, base894f169). Verified:gofmtclean,go vet ./...clean,golangci-lintreports 0 new issues vs the reviewed head, and the full package suite passes.BLOCKING
freestate byte now share one lock hold; only the twoSynccalls run outside it. A concurrentconsumerSlotcan no longer pop a slot in the fsync gap and have its new entry overwritten withconsumerFree. Regression:TestConcurrentConsumerCreateDeleteKeepsRAMDiskConsistent(64 concurrent create/delete pairs, RAM live set == reloaded set).acquirevalidates the stream name once — it is the funnel for every request path, so./..can no longer createconsumers.meta/cursors.metaoutside a stream directory. Regression:TestPublishRejectsInvalidStreamName(asserts no slot files instreams/or the storage root).+1boundary probe is only added to a positive limit, solimit == 0(no cap) returns every stream instead of a one-name page. Regression:TestListStreamsUncappedBothBackends(durable + in-memory).IMPORTANT
rename+ fsyncstreams/+ best-effortRemoveAll), matchingjournal.remove, so a crash mid-delete never leaves a half-deleted live stream.closedis set underj.muand the rename +RemoveAllstay under the same hold;appendPublishLockedreturns an error whenclosed, so a publish racing a delete cannot recreate the stream directory viarollLocked'sMkdirAll. Regression:TestDeleteRacingPublishDoesNotRecreateStream.minWatermark/minWatermarkSlot), seeded on load and invalidated by create/delete and by a raise of the minimum's slot — the every-publish/ack trim reads one value instead of sweeping up to 65 535 consumers. Regression:TestMinLiveWatermarkTracksCreateDeletePut.lastTS(offset 28) together with the count, sobudgetSnapshotorders global eviction by a segment's real end rather than its first-record timestamp. Regression:TestSealedSegmentHeaderFinalizesCountAndLastTS.MINOR (review description)
streamExistsdoc comment removed;journal.next()(kept alive only by a test) deleted.consumerStore.loadrejects empty names and duplicate cursor slots; the sanitizing slot writes are fsynced before load returns, so a freed slot is durably zero before reuse.scanSegmentbounds the frame allocation by the remaining file size and treats sub-minimum or oversized tails as torn (truncated) rather than allocating up to 4 GiB. Regression:TestScanSegmentTruncatesGarbageLength.storageBytesbounds the stream count under lazy residency.Server.Closenilss.storeunder the server lock (matching the read inacquire);store.openStreamrefuses registration oncestore.closed, so a load racing close cannot leak a journal. Regression:TestOpenStreamStateAfterCloseRefusesRegistration.listResponseOverheadcomment now documents the exact 9 + 1 accounting (the reviewer's "5" missed the response envelope: code + message-length + count + hasMore).listStreamspages with a bounded max-heap (O(n log limit)) instead of sorting the whole directory per page.Note: this Forgejo build still exposes no API/MCP action to set
resolver=trueper thread (the resolve routes return 405), so the eight threads should be marked resolved in the UI; code-wise they are all addressed by this commit.