WebStream: a Go streaming message bus with retention and client-side deduplication
  • Go 99.8%
  • Makefile 0.2%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-05 07:56:49 +00:00
.forgejo/workflows ci(code-review): run reviews on melious/deepseek-v4.1-flash-1017 with high thinking 2026-11-03 21:00:18 +00:00
.gitignore fix: resolve the phase 6 review findings 2026-11-05 07:55:32 +00:00
.golangci.yml feat: setup build 2026-09-13 20:15:25 +02:00
.gremlins.yaml test: add the gremlins mutation-testing config and record its baseline 2026-10-12 18:14:01 +00:00
acceptance_test.go docs: tighten the new code comments 2026-10-20 15:30:12 +00:00
AGENTS.md feat: replicate stream metadata on the cluster log (plan.md Phase 6) 2026-11-01 17:08:28 +00:00
bench.log fix: gate the promotion epoch on catch-up and bound placement plans 2026-10-21 10:20:34 +00:00
bench_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
conn.go docs: trim the comments this branch added 2026-11-03 23:32:52 +00:00
conn_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
consumer.go docs: tighten code comments across the package 2026-10-19 21:46:01 +00:00
consumer_state.go docs: tighten code comments across the package 2026-10-19 21:46:01 +00:00
consumer_state_property_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
consumer_state_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
consumer_test.go style: clear every golangci-lint finding in the tree 2026-10-29 06:48:27 +00:00
coverage.log fix: apply history below a copy's term instead of refusing it 2026-11-03 20:06:54 +00:00
discovery_test.go docs: trim the comments this branch added 2026-11-03 23:32:52 +00:00
doc.go docs: trim the comments this branch added 2026-11-03 23:32:52 +00:00
fuzz_test.go feat: replicate stream metadata on the cluster log (plan.md Phase 6) 2026-11-01 17:08:28 +00:00
gater.go docs: tighten code comments across the package 2026-10-19 21:46:01 +00:00
go.mod test: add rapid property models for memlog and the journal differential 2026-10-12 07:08:48 +00:00
go.sum test: add rapid property models for memlog and the journal differential 2026-10-12 07:08:48 +00:00
integration_test.go fix: drop the cached stream handle on every delete the server answered 2026-11-03 22:41:44 +00:00
journal_property_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
lazy_residency_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
LICENSE feat: initial commit 2026-09-13 18:14:31 +02:00
Makefile feat: replicate stream metadata on the cluster log (plan.md Phase 6) 2026-11-01 17:08:28 +00:00
membership.go fix: resolve the phase 6 review findings 2026-11-05 07:55:32 +00:00
membership_cluster_test.go feat: replicate stream metadata on the cluster log (plan.md Phase 6) 2026-11-01 17:08:28 +00:00
membership_join_test.go test: cover the leader a write no longer promotes over, and the promote a removed node cannot send 2026-11-01 14:20:51 +00:00
membership_test.go fix: resolve the phase 6 review findings 2026-11-05 07:55:32 +00:00
memlog.go docs: tighten code comments across the package 2026-10-19 21:46:01 +00:00
memlog_property_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
memlog_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
mesh_bench_test.go feat: holder sets and R < N placement (plan.md Phase 4) 2026-10-20 16:16:16 +00:00
mesh_test.go docs: trim the comments this branch added 2026-11-03 23:32:52 +00:00
message.go feat: server side deduplication 2026-09-15 09:41:34 +02:00
metadata.go fix: resolve the phase 6 review findings 2026-11-05 07:55:32 +00:00
metadata_property_test.go docs: trim the comments this branch added 2026-11-03 23:32:52 +00:00
metadata_test.go fix: resolve the phase 6 review findings 2026-11-05 07:55:32 +00:00
no_copy.go feat: initial commit 2026-09-13 18:14:31 +02:00
node_budget_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
placement.go docs: trim the comments this branch added 2026-11-03 23:32:52 +00:00
placement_property_test.go feat: holder sets and R < N placement (plan.md Phase 4) 2026-10-20 16:16:16 +00:00
placement_test.go docs: trim the comments this branch added 2026-11-03 23:32:52 +00:00
plan.md docs: phase 5 records what the review found 2026-11-01 14:20:51 +00:00
promote.go docs: trim the comments this branch added 2026-11-03 23:32:52 +00:00
promotion_test.go style: clear every golangci-lint finding in the tree 2026-10-29 06:48:27 +00:00
property_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
protocol.go feat: replicate stream metadata on the cluster log (plan.md Phase 6) 2026-11-01 17:08:28 +00:00
protocol_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
quorum_property_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
quorum_test.go feat: replicate stream metadata on the cluster log (plan.md Phase 6) 2026-11-01 17:08:28 +00:00
README.md fix: gate the promotion epoch on catch-up and bound placement plans 2026-10-21 10:20:34 +00:00
recovery_test.go style: clear every golangci-lint finding in the tree 2026-10-29 06:48:27 +00:00
replicate.go fix: resolve the phase 6 review findings 2026-11-05 07:55:32 +00:00
replication_consumer_test.go style: clear every golangci-lint finding in the tree 2026-10-29 06:48:27 +00:00
replication_test.go feat: replicate stream metadata on the cluster log (plan.md Phase 6) 2026-11-01 17:08:28 +00:00
rpc_stream.go docs: tighten code comments across the package 2026-10-19 21:46:01 +00:00
server.go fix: resolve the phase 6 review findings 2026-11-05 07:55:32 +00:00
server_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
store.go fix: keep a config change's writer leading until it is confirmed 2026-10-28 09:12:06 +00:00
store_group_commit_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
store_test.go docs: trim the comments this branch added 2026-11-01 13:13:39 +00:00
stream.go docs: tighten code comments across the package 2026-10-19 21:46:01 +00:00
stream_state.go feat: holder sets and R < N placement (plan.md Phase 4) 2026-10-20 16:16:16 +00:00
stream_state_property_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
stream_state_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
stream_term_property_test.go fix: resolve the phase 6 review findings 2026-11-05 07:55:32 +00:00
stream_term_test.go fix: resolve the phase 6 review findings 2026-11-05 07:55:32 +00:00
stream_test.go test: drop comments from tests 2026-10-19 22:00:54 +00:00
term.go docs: improve documentation 2026-09-14 21:18:31 +02:00
term_test.go feat: server stream handling 2026-09-13 22:03:14 +02:00
TESTING_PLAN.md feat: holder sets and R < N placement (plan.md Phase 4) 2026-10-20 16:16:16 +00:00

webstream

Simple and distributed in-process message queue system inspired by NATS JetStream powered by libp2p.

Properties

  • Server mesh is managed by libp2p.
  • No subjects.
  • Streams and consumers are typed.
  • Streams implement InterestPolicy and DiscardOld policies only.
  • Consumers are always durable.
  • All ACKs are synchronous (DoubleAck).
  • Pull consumers only.
  • Consumer handlers return errors. No error = double ack, regular error = nak, typed Term error = term.
  • Client<->server communication uses WebTransport for browsers (Go WASM) or raw QUIC streams for Go native client (using libp2p streams).
  • Communication between server nodes uses Go native client instead of custom wire protocol.

What webstream is

A library that embeds a Server and/or a client API (Conn / Stream[T] / Consumer[T]). Client and server can share one process, but that is only one deployment mode: a Go-native client can equally dial a remote webstream server, and a browser WASM client can dial the same server over WebTransport. All communication — client↔server and, later, server↔server — is libp2p application streams on a custom protocol. The Server is a libp2p host; a Conn opens streams to it with host.NewStream.

A server host listens on multiple libp2p transports at once and serves all client kinds simultaneously: QUIC for Go-native clients (in-process or remote, including server-to-server replication) and WebTransport for browser WASM clients. Which transport a client uses is a dial-address concern; the protocol handler and client code are identical.

Deployment / client modes

  1. In-process client. One Go process, one Server, and a Go-native Conn dialing the server over a loopback libp2p stream. All state in memory.
  2. Remote Go-native client. The same Conn dials a server in another process over QUIC.
  3. Server mesh. Multiple Server nodes form a libp2p mesh, each holding a replica of every stream. Each node is both server and client: it uses the same Go-native Conn API over libp2p streams to sync state with peers. There is no separate replication wire protocol.
  4. Browser client (later). A Go WASM (or js-libp2p) app uses the same Conn client API and the same application protocol over WebTransport.

How the design uses libp2p

These facts drive the architecture:

  • WebTransport is a transport configured on the libp2p host (/ip4/.../udp/0/quic-v1/webtransport, default in DefaultTransports), not a separate application layer. QUIC is likewise a host transport.
  • The application protocol (SetStreamHandler on the server, host.NewStream on the client) is transport-independent. The same protocol handler serves native QUIC peers and browser WebTransport peers; libp2p negotiates the transport under the hood.
  • A single host can listen on several transports at once. Go's DefaultTransports and DefaultListenAddrs include TCP, QUIC-v1, and quic-v1/webtransport, so one Server serves native and browser clients simultaneously with no per-client configuration.
  • Browser WebTransport dials need the server's /certhash/... on the multiaddr (AddCertHashes), but that is an addressing/embedder concern, not an API difference.
  • libp2p refuses to dial its own peer ID (swarm.ErrDialToSelf), so the in-process loopback client cannot reuse the server host. It owns a libp2p host of its own and dials the server over a loopback address; the two hosts share one process but are otherwise exactly like a remote client.
  • Therefore: no custom transport seam is needed. Conn/Stream/Consumer are written directly against p2pnetwork.Stream; native clients dial a QUIC multiaddr and browser clients dial a WebTransport multiaddr with the same code.

Current state

The single-server milestone is implemented — durable when WithStore is configured — and the full verification suite passes:

go build ./...        # ok
go vet ./...          # ok
golangci-lint run     # 0 issues
go test ./...         # ok
go test -race -count=1 ./...  # ok

The client/server split the design calls for is real: the server side is untyped, non-generic in-memory state (streamState, consumerState) plus request handlers reached only through libp2p streams, and the client side is typed generic handles (Conn, Stream[T], Consumer[T]) that hold only codecs plus remote identifiers. Framing is length-prefixed and WASM-portable, the double-ack pull model works end to end, and ack order, resource limits and the allowlist are enforced server-side.

Milestone scope:

  • Disk persistence is in: WithStore(*os.Root) makes the server durable — records, sequence bookkeeping and consumer progress survive a restart. A stream is a self-contained directory: a segmented record log plus two fixed-stride consumer slot files. Without it, all stream/consumer state lives in server memory, unchanged.
  • Retention is in: streams apply InterestPolicy retention with DiscardOld discard, bounded by a per-stream maxAge window (WithMaxAge) and the server's storage budget (WithStorage, defaulted to 256 MiB), which is a global pool — messages are evicted only when all streams combined exceed it.
  • Producer-side deduplication is in: a publish carrying a WithDeduplicationID is idempotent, and Stream.Publish reports through PublishAck.Duplicate whether the server already held the record. The dedup window is retention itself, so it needs no timer and no configuration.
  • Consumer-side deduplication is in: delivery stays at-least-once, and the consumer skips (but re-acks as a skip) a record it has already processed, for exactly-once processing. The watermark lives in the server's consumer state and comes back with the handle, so the client stays stateless.
  • A server mesh is in, for records and consumer state: WithMeshPeers(ids...) makes a node one of a set that each holds a replica of the streams it is placed on. Placement is derived, never configured: R = min(3, N) and the holders are the R lowest-ranked live nodes, so the leader is always the first holder and a deployment of three nodes or fewer keeps R = N. One node leads each stream: the lowest-ranked node by hash(stream name, peer ID) among the peers the failure detector currently believes are running. It serves every write for the stream; a node that is not the leader relays the request to the leader, and a record read is served locally on any holder and relayed on any other node to whatever node routing names — the live leader, otherwise the placement candidate, which is a live holder. Leadership is a term the leader announced after a read quorum (Server.Promote), not a fact derived from the set: every node relays to the leader its stream's latest term announcement named while that node answers, and to the placement rule's candidate when it does not. The leader writes each record to a hidden command log (internal-<name>) before applying it to its own copy, and every holder tails that log and applies the same commands, so the holders hold the same records, sequence numbers and timestamps. Retention is the leader's decision and travels as an explicit command, so a replica's window matches the leader's. Consumer state is stream state and replicates on the same command log: a consumer's watermark and deletion travel as explicit commands, so a consumer created or acked on one node is visible on every other with the same watermark, and a deleted consumer does not resurrect. One consumer name has one live consume loop across the mesh: a pull on a non-leader validates its lease on the leader on every pull (one extra hop; the record read stays local) and serves the record at the leader's cursor, and consumer-metadata reads are forwarded to the leader, so a restarted handle resumes exactly where it stopped no matter which node serves it.
  • The holder set is a replicated record, and placement changes are ordinary. The leader writes Holders{epoch, ids} on the stream's own command log; that epoch is the set every write quorum and every promotion's read quorum is taken over, so two nodes whose live views disagree still count the same votes. A node that is not named by the epoch it applied does not serve as a holder. A failure re-places the stream on the next-ranked live peer, and a joining holder is admitted by the epoch only after its replica consumer has applied the leader's retained window — so a placement change never has a moment with no holder. Placement moves one stream at a time, and Server.Placements reports the epoch, holders and in-progress set per stream. R is not configurable (WithReplicationFactor does not exist): an R-of-N deployment's read availability is at least one holder live, and its write availability is at least quorum(R) holders live.
  • A publish on a mesh leader is acked only once its record is durable on a quorum of the stream's holders — quorum(R) = R/2 + 1, the leader included — the confirmation being the ack a replica sends after applying the command durably. An ack therefore means "a majority holds this record", which is what promotion checks; a publish that cannot reach the quorum within quorumTimeout fails with ErrNoQuorum and changes nothing about the record, which is already durable on the leader.
  • Failover is in, and it is a promotion, not an election. A per-peer heartbeat on the existing control connection (plus the failure of any other request on it) keeps a live view with a three-interval lease. When the stream's leader is gone from that view, the first write that reaches the placement rule's candidate promotes it (Server.Promote, also callable directly): it takes a read quorum of the stream's recorded holder set, durably raises the stream's term above every term it can observe, copies any committed record its copy lacks, rebuilds the command log so the replicas converge on what it holds, and only then serves. A promotion that cannot reach a majority fails with ErrReadQuorum (which matches ErrNoQuorum) and serves nothing. A leader that comes back — or that never heard of the promotion — is fenced by the term: it refuses the write with ErrStaleTerm, follows the new leader and converges by replay. Server.TransferLeadership hands a stream to the next live holder for planned work. A node serves a stream only at a term it announced itself; a copy loaded from disk is not serving until it promotes.
  • The server host serves native and browser transports at once; this milestone only exercises the loopback native path but does not hard-code a single transport.
  • Through a stream, publish messages and create/manage consumers.
  • Through a consumer, register message handlers via Consume(ctx, handler HandleFunc[T]).
  • Values are typed on the client via EncodeFunc[T] / DecodeFunc[T]; the server only ever sees opaque bytes.

Out of scope (deferred): WebTransport/client configuration itself, dynamic membership (R follows the member count, but the count does not change yet), batched fetches, and compaction. Disk persistence is implemented: records, sequence bookkeeping and consumer progress survive a restart, and the stream directory (streams/<name>/ with stream.meta, log/, consumers.meta and cursors.meta) is the unit of storage a replica copies. Compaction is not needed at this point: trimming only ever advances the retained base and journal.trimTo deletes whole sealed segments below it, so the log never develops interior holes. The remaining mesh work is specified in plan.md.

Testing

The verification suite is layered; TESTING_PLAN.md is the inventory and rationale of every layer.

make test        # go test -race ./...
make test-props  # + the durable journal differential (go test -tags rapid ./...)
make cover       # coverage.out + total (gremlins' profile)
make cover-props # same, with the rapid tag
make bench       # full benchmarks, raw output to bench.txt (gitignored)
make bench-log   # appends the stable micro-benchmarks and the quorum ones to the committed bench.log
make fuzz        # runs every fuzz target for FUZZTIME (default 10s)
gremlins unleash # mutation testing, configured by .gremlins.yaml

Layers:

  • Native fuzz targets for every byte-level parser (request frames, on-disk segments, stream names), with f.Add seeds so plain go test replays them and make fuzz explores them.
  • Rapid property models (test-only dependency) as the executable spec of the storage layer: the memlog append/recover/trim machine, the on-disk journal differential, consumer state (leases, cursors, watermarks) and the streamState publish/consume lifecycle (dedup table tied to retention, the interest-vs-age trim point, the cached minimum cursor). They run on every go test; only the durable journal differential — tens of seconds of real disk I/O, one fsync per append — is gated behind the rapid build tag.
  • Deterministic time: time-dependent tests run inside testing/synctest bubbles with a fake clock (via rapid.SyncTest in the models). No real-time sleeps, no field backdating, and no test seams in production.
  • In-memory transport: mesh, conn, consumer and integration tests run on libp2p's mocknet instead of real TCP, so their blocking is durable inside bubbles. Only the benchmarks and the QUIC transport test use real transports (they measure them).
  • Mutation testing: .gremlins.yaml configures gremlins; its dry run is the NOT COVERED sentinel. Case-guard, const-arithmetic and heap-interface NOT COVERED findings are known false positives.
  • Regression tracking: committed coverage.log and bench.log record the coverage total and the machine-stable micro-benchmarks per meaningful change.

Resolved decisions

Decision Choice
Deployment one Server host; client may be in-process loopback (own host, two hosts in one process), remote Go-native, or (later) browser WASM
Transport libp2p application streams over the host; the host listens on QUIC (native) and WebTransport (browser) simultaneously
WebTransport a host transport, configured later; serves the same protocol, no API change
Encoding hand-rolled length-prefixed binary (encoding/binary), portable to JS/WASM
Delivery JetStream-style pull consumer: client pulls, handler processes, ack is confirmed synchronously (double ack); one message in flight per consumer
Consumer pacing Consume is event-driven and drains back to back: a pull blocks on the server until a record is available, delivered records trigger an immediate re-pull, and defaultConsumeInterval (1 s) only paces retries after an empty/failed pull or a nacked redelivery
Topology one control stream for RPCs; one dedicated pull stream per active consumer
Frame-size boundary one maxFrameSize (1 MiB) cap on frame payloads in both directions; the pull response overhead (recordDataOverhead) and the publish request overhead (publishFrameOverhead) are costed in, so a legally published record is always pullable (maxRecordSize); oversized publishes are rejected locally with ErrRecordTooLarge before any byte reaches the transport, and an oversized request fails only that request, never the connection
Singleflight coalesced CreateStream/CreateConsumer run the shared RPC under a server-owned context (context.WithoutCancel + defaultFlightTimeout), never a caller's; each caller still waits on its own ctx
Write bounds every request write is bounded: the caller's deadline is used when it is near, otherwise the write falls back to defaultWriteTimeout (30 s), so context.Background()/long-lived callers cannot hang a Consume shutdown on a stalled transport
Dedup / disk producer- and consumer-side deduplication are implemented (below); disk persistence is implemented (WithStore, see the Persistence row)
Persistence WithStore(*os.Root) makes the server durable. The stream is the unit of storage: each stream is a self-contained directory <root>/streams/<name>/ holding a stream.meta (maxAge), a segmented record log under log/ (files named by each segment's base sequence number, 20-digit zero-padded) and two fixed-stride consumer files (consumers.meta, cursors.meta). Names are 1..64 bytes of [a-zA-Z0-9-], so a name is a safe path component with no encoding step. Without WithStore the server is purely in-memory, unchanged
Durable publish ack a publish's record is fsync'd before its ack returns, so an acked record survives a crash, and fsyncs are group-committed: one fsync covers every record written before it started, publishers admitted while one is in flight are covered by the next one, and every writer in a batch is acked when that fsync returns. There is no interval to configure — an idle server pays exactly the one fsync per publish it always has, a busy one pays about one per batch, and there is no crash window to trade away. Appends to the segmented log are also the only writer of it, so on-disk order is sequence order
Write quorum an acked publish is durable on a quorum of the stream's holders, quorum(R) = R/2 + 1 with R = min(3, N) the holder count, so 1/2/3 holders need 1/2/2. The quorum is derived from R, never configured: a promotion takes a read quorum of the same set, and the two majorities must intersect. The confirmation is the replica consumer's existing ack — a replica acks only after applying durably — so the leader waits on a channel the replication path already owns, and a relayed ack is counted only for the node that sent it (its consumer name is its own peer ID), so a vote cannot be cast for a node that never applied the record. The leader's own copy is the vote its append already made durable. A publish that cannot reach the quorum before quorumTimeout fails ErrNoQuorum — and so does a wait the caller cuts short, with the cancellation as its inner error — and is not undone: the record may be readable on the leader and may replicate later, so a retry that must not duplicate carries a WithDeduplicationID and reads the original sequence number back (a duplicate reply is an ack for the same sequence, so it waits on the same quorum). A node that relays a publish on a stream it does not lead relays it on a connection of its own: a node serves one application stream's requests one at a time, so a relayed publish waiting out its quorum on the shared per-peer connection would stall that node's own reconciliation and replication registration — the very work that produces the confirmation
Durable consumer state consumer progress lives in the two slot files: consumers.meta maps a consumer name to a cursors.meta slot whose double-buffered, CRC-protected watermark is updated in place per ack. Only the watermark is durable — the cursor is derived as max(watermark, base) and never stored or sent on the wire. Watermark writes are coalesced write-behind, not fsync'd per op: a lost watermark redelivers one already-processed record — the at-least-once edge the consumer's watermark then skips — while a lost record would not be recoverable. Creating a consumer is self-healing (its only durable state is watermark 0); deleting one zeroes its cursor slot and fsyncs it before freeing the index entry, so a deleted consumer never resurrects and a recreated one never inherits a high watermark. lease/leaseUntil are runtime-only, so a recovered consumer starts unleased
Recovery on startup WithStore reads each stream directory, derives the window from the segment set, replays the record frames and scans the consumer slots, restoring the retained window, the dedup map, the byte budget cache and every consumer's watermark. base is the oldest segment's filename number; next is the newest segment's base plus its scanned frame count, so a sequence number is never reused without persisting a counter and an emptied stream keeps its newest segment and continues numbering. A torn tail is truncated; a wrong magic/version or a corrupt mid-file frame refuses that stream rather than misreading the rest. A deleted stream's directory is removed (tombstone + RemoveAll) so it does not resurrect, and a cursor the recovered window has outrun is repaired on the next pull (repairCursor)
nats-io/nats.go/jetstream removed (was reference-only)
Retention fixed InterestPolicy: a record is retained while at least one consumer has not acked or terminated it, so a stream with no consumers retains everything on interest; the retained window is [base, next), trims advance base only forward, and sequence numbers are never reused
Discard fixed DiscardOld: the oldest records are dropped first when a record passes the stream's maxAge window or the server's combined storage budget is exceeded, regardless of interest — a bound can trim a record a slow consumer has not acked, and that consumer's next pull skips to the retained base instead of stalling
Age window WithMaxAge(d) sets how long a stream retains a record; older records are discarded even with no consumer activity, driven by a per-stream timer armed for the oldest record's expiry. Zero retains forever. The server is the source of truth: re-creating an existing stream is a no-op, so the window binds only at creation
Storage budget WithStorage(bytes) caps the total bytes the server retains across all streams; without it NewServer defaults to defaultStorageBytes (256 MiB) so an unconfigured server cannot be driven into unbounded memory, and WithStorage(0) opts back into unbounded. The budget is a global pool: a message is evicted only when all streams combined exceed it, and the oldest records across the whole server are dropped first until the total fits again — individual streams are never capped on their own
Trim point keep = max(minConsumerCursor, maxAgeKeep); the interest contribution is the minimum cursor across consumers and the age contribution is the first record still within its maxAge window. Trims run after append, after an ack/term advances a cursor, after a consumer is deleted, and on the age timer; the storage budget is enforced after publish by evicting the globally oldest records across streams. A trim that moves base wakes blocked pulls. memlog keeps a byte counter (log.bytes) so the global retained total is accounted without rescanning, subtracting exactly the dropped retained records and skipping the stale mid-chunk prefix left by an earlier trim
Ack order strict equality against the consumer cursor, validated in the same critical section as the move (consumerState.advanceIf, a compare-and-set); skipped, duplicate and MaxUint64-overflow acks/terms/naks are rejected with ErrAckOutOfOrder (codeAckOutOfOrder), a per-request error that leaves the stream usable; ErrConsumerNotFound is checked first
Error classification every wire code maps to an exported sentinel matchable with errors.Is (ErrInvalidRequest, ErrInternal, ErrLimitExceeded, ErrRecordTooLarge, ErrAckOutOfOrder, ErrStaleTerm, ErrNoQuorum, plus the existing not-found sentinels); truncated/undecodable request payloads are codeInvalidRequest; codeInternal replies are masked with a generic message and the real error is logged server-side
Resource limits server caps names (maxNameSize, 64 bytes), streams per server (maxStreamsPerServer), consumers per stream (maxConsumersPerStream) and concurrent application streams per peer (maxStreamsPerPeer); over-limit creates fail with ErrLimitExceeded; get-or-create on existing names stays a no-op at a limit
Terminal consumer errors the consume loop stops on a deleted stream/consumer (ErrConsumerNotFound/ErrStreamNotFound) or a broken connection (ErrConnClosed), logs the failure once, and surfaces it through Consumer.Errors(); a graceful ctx-cancelled stop closes that channel without a value; restarting Consume replaces the channel
Publish result Stream.Publish returns a PublishAck{Seq, Term, Duplicate}: the server-assigned sequence number, the stream term it was served under, and whether the server deduplicated the publish. The record timestamp is not transmitted (stays server-side in memlog.record)
Dedup ID WithDeduplicationID encodes the empty ID as an empty bytes field and caps at the frame payload cap (no MaxUint16/empty special-casing); an empty ID never dedups
Producer dedup a publish carrying a deduplication ID is idempotent (exactly-once storage): the server keeps a per-stream msgID → seq table and answers a repeat with the original sequence number and duplicate: true without appending. The lookup, append and mapping share one critical section (streamState.appendDedup), so two racing retries of one ID cannot both store
Dedup window tied to retention, not a fixed timer or a magic constant: an entry is dropped by the same trim that drops its record, so fast consumers give a short window and slow or absent ones a long one, and the table is bounded by the retained records carrying an ID. A repeat that arrives after its record was trimmed is outside the window and stored again under a new sequence number; a repeat while the record is retained returns the original sequence number even if it has already been delivered. The ID is held on the record it identifies, so it counts against the storage budget like the payload (record.retainedBytes) and the trim that drops the record drops its entry in O(records dropped) rather than sweeping the whole table
Delivery guarantee an ack confirms storage, not delivery; delivery stays at-least-once (a nak or an ack lost in flight redelivers), so the consumer deduplicates by sequence number
Consumer dedup one watermark per consumer (processedBefore, exclusive) is enough because processing is strictly sequential — one message in flight, cursor advances only on a confirmed ack or term — so no set or interval is needed (revisit when batched fetches or parallel processing land). A delivered record below it is a redelivery whose ack was lost: the handler is skipped and the record re-acked as a skip, since skipping without re-acking parks the cursor and redelivers forever. A skip advances the cursor and watermark exactly like an ack but is distinguishable server-side for observability. Only ack, term and skip raise it — a nak is the intended retry and must re-run the handler — and it is raised when the handler returns, before the ack is known to have landed, because the duplicate worth preventing is running the handler again. Exactly-once processing holds for every record whose ack the server applied; a crash between the handler finishing and the ack being applied can reprocess that one record
Consumer progress lives on the server (the durable processedBefore watermark, with the cursor derived as max(watermark, base)), never on the client's disk: the client is stateless and a future mesh replica reads the same state. A cursor repaired forward past discarded records does not raise the watermark — those records were never processed. Stream.Consumer/Stream.CreateConsumer return it with the handle, and a refresh only ever raises the client's watermark, since a lost ack leaves the server behind the handle
Single-loop consume at most one consume loop may run per consumer, enforced server-side by a consume lease. Every pull carries the loop's token (pullRequest.lease); a pull by a different token while the lease is fresh fails with ErrConsumerBusy (codeConsumerBusy), so a second Consume is rejected. The lease is refreshed on every pull and expires consumerLeaseTimeout (30 s) after the last one, so a crashed loop (whose pulls stop) frees the consumer to a replacement without any explicit signal — the "misses some expected pulls" backstop. A graceful shutdown sends an explicit release (frameConsumeRelease) so a replacement acquires immediately. A zero lease opts out (administrative/direct-protocol pulls). Trade-off: because one message is in flight at a time, the window must also exceed the time a record spends in the handler; a handler blocked past it can let a second loop take over
Response shape the response frame echoes the request kind, so response is encoded and decoded per kind: only the fields that kind defines are on the wire (publish: seq + duplicate; pull: the optional record; list: the sorted page of names + hasMore; create/get consumer: the processedBefore watermark; the rest: nothing). code and message are common and a failure reply stops after them, so a publish ack carries no record, list or watermark bytes
Stream / consumer lookups Conn.Stream and Stream.Consumer are wire operations, not local cache reads: the server answers whether it still holds the stream (dropping a stale client handle) and returns the consumer's progress. A stream handle still comes from the local cache because it carries the codecs; a consumer needs none, so Stream.Consumer rebuilds a handle a restarted client has lost
Placement R = min(3, N) and the holders are the R lowest-ranked live nodes, derived rather than configured (there is no WithReplicationFactor), so the leader is always the first holder and every deployment of three nodes or fewer is unchanged. The leader writes Holders{epoch, ids} on the stream's own command log, and that epoch — not a fresh recomputation — is the set a write quorum and a promotion's read quorum are taken over, so nodes whose live views disagree still count the same votes. A holder set never shrinks below R — a dead holder is replaced by the next-ranked live peer, not simply dropped, so a publish is never acked on fewer copies than the member count calls for — and a joining holder is named by an epoch only after its replica consumer has applied the leader's retained window; a node the epoch drops releases its copy and its rep-<peerID> consumer, so the command log unpins. A record read is local on every holder (read availability is the holder set's, not the leader's) and is forwarded one hop on a non-holder, to the live leader or, when it no longer answers, to the live holder the placement rule names — which retires the shipped "record reads are never forwarded" rule
Availability stated from the derived rule, never from a knob: reads need at least one holder live, writes need at least quorum(R) holders live. N = 1 is the single-node server unchanged; N = 2 keeps durability (one surviving copy) and reads but loses writes the moment either node dies, because quorum(2) = 2; N ≥ 3 (R = 3, quorum = 2) tolerates one failure for writes and two for reads. Two holders buy a second copy and no failover — the trade-off a two-machine deployment is making
Security no auth, no per-connection scoping (documented); ack order enforced server-side; NewAllowlistGater (host connection gater) and WithAllowedPeers (stream handler) allowlisting as hardening before untrusted exposure

Target architecture

A Server is one libp2p host that registers the protocol once and listens on several transports at the same time (QUIC for native, WebTransport for browser).

  Go-native client (in-process or remote)        browser WASM client
 ┌───────────────────────────┐                  ┌────────────────────────┐
 │ Conn                      │                  │ js-libp2p / go-wasm    │
 │  └─ Stream[T]             │                  │ same libp2p streams    │
 │      └─ Consumer[T]       │                  └───────────┬────────────┘
 │           handler(msg)    │                              │
 └─────────────┬─────────────┘                              │
               │ host.NewStream(serverID, protocol)         │
               │        QUIC transport                      │ WebTransport
               ▼                                            ▼
 ┌────────────────────────────────────────────────────────────────────┐
 │ Server libp2p host — SetStreamHandler(streamProtocol) once         │
 │   listen: /ip4/0.0.0.0/udp/0/quic-v1                               │
 │           /ip4/0.0.0.0/udp/0/quic-v1/webtransport (+ certhash)     │
 │   streams:   map[name]*streamState    (in-memory log + cursor)     │
 │   consumers: map[key]*consumerState                                │
 └────────────────────────────────────────────────────────────────────┘

The same Conn dials a remote peer's multiaddr for mesh replication, and a browser's libp2p stack dials the WebTransport multiaddr; client code is unchanged. In-process loopback is the same Conn in the same process pointed at the server host's peer ID; because libp2p refuses to dial its own peer ID (swarm.ErrDialToSelf), the client uses a host of its own and the two hosts are connected over a loopback address, exactly as the integration tests arrange it.

  • Control channel: request/response RPCs for create/delete stream and consumer, publish, info/list. Correlated by request ID.
  • Pull channel (per consumer): client issues pull requests; server returns the next record at the consumer's cursor; client decodes and runs the handler; client sends the result (ack/nak/term); server confirms, then advances or redelivers. One message in flight per consumer → synchronous double ack.
  • Server log: ordered records {seq, ts, dedupID, data} held in memory and, with WithStore, appended to a segmented log in the stream's directory. A per-consumer cursor plus the InterestPolicy/maxAge/storage-budget rules trim the retained window [base, next).
  • Protocol ID: streamProtocol (/norobo/webstream/1.0.0); keep it stable so browser and native peers agree.
  • Transport-agnostic handlers: handlers operate on p2pnetwork.Stream only; nothing in the protocol path branches on QUIC vs WebTransport.
  • Server-side hardening: a connection-gater allowlist (NewAllowlistGater) and a stream-handler allowlist (WithAllowedPeers) gate who can reach the protocol; acks are validated against the consumer cursor in one critical section (advanceIf) and rejected out of order; resource limits (maxNameSize, maxStreamsPerServer, maxConsumersPerStream, maxStreamsPerPeer) bound what one peer can create.

Notes / assumptions

  • Consume keeps its void signature; transport errors during pull are surfaced through c.logger. A terminal failure — a deleted stream or consumer, a broken connection, or a pull stream that could not be opened — stops the loop, is logged once, and is delivered through Consumer.Errors(); a graceful ctx-cancelled stop closes that channel without a value. Restarting Consume swaps in a fresh channel.
  • Pull fetches one record at a time this milestone; batching can be added later without changing the frame kinds.
  • The loopback milestone runs two libp2p hosts in one process because libp2p refuses self-dial; the server can later be created with the WebTransport listen addr and certhash advertisement for browser clients, with no change to Conn.
  • The frame-size boundary is enforced on the payload in both directions: Publish is bounded by the pull response cap (maxRecordSize) and by the publish request frame, so an over-cap publish fails locally with ErrRecordTooLarge and never breaks the connection or leaves a record unpullable.
  • Coalesced create operations run under a server-owned context; a caller's cancellation only aborts its own wait, never the shared flight.
  • Every request write is deadline-bounded (context deadline when near, defaultWriteTimeout otherwise), so a stalled transport cannot hang Consume shutdown.
  • Trust model: names are global across peers and there is no per-connection scoping (documented). Ack order is enforced server-side, and the connection gater / stream-handler allowlist is the hardening to deploy before exposing the server to untrusted peers.