fix: stop pooling request waiters — a borrow safe only by fragile ordering #9

Merged
sam merged 1 commit from fix/request-waiter-pool into feat/mesh-replication-command-log 2026-10-04 07:55:35 +00:00
Member

Scope

Follow-up to PR #8 comment 758, which claimed a pooled-wireWriter data race in the client. This PR corrects that claim and fixes the one genuine fragility the investigation surfaced.

This PR stacks on feat/mesh-replication-command-log (PR #8).

The correction

Comment 758 said the client's pooled wireWriter "can hand the same writer to concurrent requests on different streams", corrupting frames. Testing that hypothesis disproved it:

  • The sync.Pool handoff is exclusive: instrumenting getWaiter/borrow lifetimes showed no waiter was ever handed to two live requests across thousands of borrows.
  • The apparent corruption was a test artifact: the stress assertion ack.Seq == value is invalid under concurrency because the server assigns sequence numbers in arrival order — publish 350 answered with seq 0 means 350's request arrived first, not that responses crossed.
  • The race-detector reports around rpcStream are false positives of the pure-Go detector (the only detector available here — no CGO): it flags r.pending[requestID] = waiter and r.pendingLock.Unlock() — both inside the lock's own critical section — and even pairs accesses to different heap addresses as racing. The same detector reports (*Server).acquire's rs.refs++ racing, which is provably under s.lock.

The fix

The requestWaiter pool is a real fragility of the same borrow class, and it is the one worth removing:

  • getWaiter mutates the pooled struct's fields (kind, ch) with no lock.
  • The waiter's lifetime crosses into the reader goroutine's critical section (deliver reads waiter.kind and sends on waiter.ch under pendingLock).
  • The pool is safe today only because two unenforced orderings happen to hold: deliver notifies under pendingLock, and every pool return follows the pending-map removal. Reorder either and the borrow goes stale.
  • The pool saves one small allocation per request. Removed; one waiter per request now.

The wireWriter pool is kept: its borrow is strictly synchronous inside request (the write completes before the put), so it cannot hand the same buffer to two live borrowers.

Test

TestRPCStreamConcurrentRequests: 8 goroutines × 50 publishes on the control stream while a consumer loops on its own pull stream (pulls + acks in flight on a second rpcStream), asserting the real identity contract — every publish answered exactly once, response seqs exactly 0..N-1 once each. This is the regression test a waiter mixup would break (a lost or duplicated response shows up as a missing or duplicated seq).

Verification

  • go build ./..., go vet ./..., gofmt -l . clean.
  • go test ./... clean ×4; stress test 10/10.
  • racedetector on the mesh/replication/conn subset: clean (0 reports).
  • golangci-lint run --fix=false: 81 issues — identical to main's baseline; nothing in the edited files.
  • go.mod/go.sum unchanged.
## Scope Follow-up to [PR #8](https://git.norobo.chat/norobo/webstream/pulls/8) comment [758](https://git.norobo.chat/norobo/webstream/pulls/8#issuecomment-758), which claimed a pooled-`wireWriter` data race in the client. This PR corrects that claim and fixes the one genuine fragility the investigation surfaced. **This PR stacks on `feat/mesh-replication-command-log` (PR #8).** ## The correction Comment 758 said the client's pooled `wireWriter` "can hand the same writer to concurrent requests on different streams", corrupting frames. Testing that hypothesis disproved it: - The `sync.Pool` handoff is exclusive: instrumenting `getWaiter`/borrow lifetimes showed **no waiter was ever handed to two live requests across thousands of borrows**. - The apparent corruption was a test artifact: the stress assertion `ack.Seq == value` is invalid under concurrency because **the server assigns sequence numbers in arrival order** — `publish 350 answered with seq 0` means 350's request arrived first, not that responses crossed. - The race-detector reports around `rpcStream` are false positives of the pure-Go detector (the only detector available here — no CGO): it flags `r.pending[requestID] = waiter` and `r.pendingLock.Unlock()` — both inside the lock's own critical section — and even pairs accesses to *different* heap addresses as racing. The same detector reports `(*Server).acquire`'s `rs.refs++` racing, which is provably under `s.lock`. ## The fix The `requestWaiter` pool is a real fragility of the same borrow class, and it is the one worth removing: - `getWaiter` mutates the pooled struct's fields (`kind`, `ch`) **with no lock**. - The waiter's lifetime crosses into the reader goroutine's critical section (`deliver` reads `waiter.kind` and sends on `waiter.ch` under `pendingLock`). - The pool is safe today only because two unenforced orderings happen to hold: `deliver` notifies under `pendingLock`, and every pool return follows the pending-map removal. Reorder either and the borrow goes stale. - The pool saves one small allocation per request. Removed; one waiter per request now. The `wireWriter` pool is kept: its borrow is strictly synchronous inside `request` (the write completes before the put), so it cannot hand the same buffer to two live borrowers. ## Test `TestRPCStreamConcurrentRequests`: 8 goroutines × 50 publishes on the control stream while a consumer loops on its own pull stream (pulls + acks in flight on a second `rpcStream`), asserting the real identity contract — every publish answered exactly once, response seqs exactly `0..N-1` once each. This is the regression test a waiter mixup would break (a lost or duplicated response shows up as a missing or duplicated seq). ## Verification - `go build ./...`, `go vet ./...`, `gofmt -l .` clean. - `go test ./...` clean ×4; stress test 10/10. - `racedetector` on the mesh/replication/conn subset: clean (0 reports). - `golangci-lint run --fix=false`: 81 issues — identical to `main`'s baseline; nothing in the edited files. - `go.mod`/`go.sum` unchanged.
The requestWaiter struct is pooled, but its borrow is sound only while two
orderings hold: the reader goroutine's deliver must notify under pendingLock,
and every pool return must follow the pending-map removal. Nothing enforces
or documents either — getWaiter mutates the struct's fields with no lock, and
the waiter's lifetime crosses into the reader goroutine's critical section —
so any reordering reintroduces a use-after-pool. The pool saves one small
allocation per request; remove it and allocate one waiter per request instead.

The wireWriter pool is untouched: its borrow is strictly synchronous inside
request (the write completes before the put), so it cannot hand the same
buffer to two live borrowers.

While verifying, the corruption hypothesis behind this change was tested and
disproven: instrumenting the borrows showed no waiter was ever handed to two
live requests across thousands of borrows, and the concurrent-publish stress
assertion that appeared to show cross-delivery (publish 350 answered with
seq 0) was wrong — the server assigns sequence numbers in arrival order, so a
response's seq is not the published value. A corrected stress test
(TestRPCStreamConcurrentRequests) now pins the real contract: with concurrent
publishes on the control stream alongside a consumer looping on its pull
stream, every publish is answered exactly once and the response sequence
numbers are exactly 0..N-1 once each.
sam merged commit 663b5990f3 into feat/mesh-replication-command-log 2026-10-04 07:55:35 +00:00
sam deleted branch fix/request-waiter-pool 2026-10-04 07:55:35 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!9
No description provided.