fix: nudge the model back to work when a turn ends on prose #32
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
sam/brainless!32
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/nudge-prose-only-turns"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
Two independent causes produced the errors you kept seeing (
adk run: openai: response output did not contain text or tool content) and made the retry solution look broken:The
<brainless-clock>block was a mid-conversationrole:"system"item. It was appended to every outgoing request insideinput, after the user message. The Responses API only sanctions system text as the top-levelinstructionsfield or a leading input item. Melious's upstream silently drops a mid-conversation system item and answers HTTP 200 with nothing usable — its own error message says so: "Likely an unsupported request parameter that the provider silently dropped." Isolated with a live probe:instructions-only 4/4 usable, clock-only 0/4, plain 2/2. Pi never hits this because it never sends mid-conversation system messages (supportsMidConvoSystemMessagesdefaults false).The runner called the provider non-streaming. Brainless pinned
StreamingModeNone; melious's non-streaming Responses path is the broken one (tool-bearing non-streaming requests came back with a reasoning item and no message 6/6 times in a probe), while streaming Responses returned usable turns 4/4 — the way pi drives melious.Each degenerate round then burned the retry budget (30 × ~1s = ~30s of dead time per round, and the chat-completions fallback failed too because it inherited the same broken request shape).
Fix
StreamingModeSSE: the provider is driven over SSE (the path it implements correctly), while the harness processes only the final aggregated event — the turn behaves exactly like a blocking call. The cache-write middleware now reads usage from the SSE terminal event, so write billing survives streaming.retryEmptyOutputalso retriesErrResponseFailed; when its budget on the Responses endpoint exhausts it tries a Chat Completions fallback for the same provider/model once (verified 6/6 against melious; both clients share the cache-write box).model X returned no usable output N times in a row (empty or failed provider responses)...instead of the raw ADK error; the original stays in the log.apiconfig option ("responses"default |"chat_completions") for providers whose Responses endpoint is absent or broken.varnamelenonNewRegistry—golangci-lint run ./...is now clean.Verification
user, turns complete (6.5s typical for a single tool round; remaining latency is melious's ~77 tok/s × the rounds the model chooses).TestACPMemoryToolResultRendersContent: acreate_memoryresult reaches the client as a rendered text content block naming the entity, type, and observation.go test -race ./...,go vet ./...pass;golangci-lint run ./...0 issues.Review of fix/nudge-prose-only-turns at
c52065e, verified against the repo's own toolchain and the ADK v2.5.0 / genai v1.72.0 module sources.Verified green (commands run in a worktree at the PR head):
go build ./...OK;go vet ./...OK;go test -race ./internal/harness/OK — all 7 new unit tests plusTestACPNudgeKeepsTurnAliveAcrossProseOnlyAnswerpass;golangci-lint run ./internal/harness/clean except the disclosed pre-existingvarnamelenonNewRegistry; no go.mod/go.sum changes in the diff (no new dependencies; govulncheck reports only pre-existing, uncalled vulnerabilities).API claims checked against module source, not memory:
model.LLM/LLMRequestshape (go doc google.golang.org/adk/v2/model.LLM);genai.Content.Parts/Role,Part.Text/Thought/FunctionCall/FunctionResponse,FunctionCall.Name/Args,FunctionResponse.Name/Response; ADK's base flow appends tool results as user content withFunctionResponseparts and no text part (internal/llminternal/base_flow.go), sohasToolWorkThisTurnmatches the real conversation shape; ADK itself re-calls thought-only turns (maxConsecutiveThoughtOnlyTurns = 10, base_flow.go), so the thought-only pass-through is safe.No blocking issues. One IMPORTANT finding and two suggestions below.
IMPORTANT — the retry and nudge budgets compose multiplicatively (registry.go:99-100): each nudge round calls
retryEmptyOutput.GenerateContentfresh, re-arming its own 30-attempt / 15s budget, so a model that alternates empty outputs and prose-only answers (both failure modes documented for melious/deepseek-v4-flash-0731, the exact model this PR targets) can cost up to 3 × 31 = 93 full-conversation re-sends and ~45s per model call. Suggest sharing one budget across both wrappers before merge.@ -661,0 +760,4 @@// The nudge must not end the turn: the tool the model called on the second// round executed (start + completed), so the turn produced real work.if len(client.toolCalls) != 2 ||🟡 SUGGESTION — the test cannot detect a wrong nudge on the final answer. The stub serves the same text ("Found it: pong.") for every request >= 3 (default case, lines 699-703), and the assertions here (2 tool calls, updates = [Let me look at the file., Found it: pong.]) are identical whether the final answer passed through cleanly or was wrongly nudged twice (the model would answer the same text again, then the budget forces a pass-through). So a regression where
hasToolWorkThisTurnalways returns false still passes this e2e. Verified: stub + assertions atc52065e; test passes with current code. Fix: count requests in the stub (it already tracksstep) and assert exactly 3 — that pins the "final answer after tool work is never nudged" claim to the real flow instead of the hand-built shape in TestNudgeToolUseSkipsWhenToolsRanThisTurn.@ -0,0 +56,4 @@func (n *nudgeToolUse) Name() string { return n.llm.Name() }func (n *nudgeToolUse) GenerateContent(ctx context.Context, req *model.LLMRequest, stream bool) iter.Seq2[*model.LLMResponse, error] {🟡 SUGGESTION — the wrapper ignores the
streamflag. Under streaming, ADK's openaimodel "yield[s] a partial per delta and one final response built from the accumulated turn" (internal/completions/model.go, generateStream), so the first pair is a partial text delta with no tool call yet:eligibleForToolUseNudgewould fire on it, the whole accumulated stream would be discarded and re-sent, andcollectModelPairsbuffers the entire stream in memory. Today unreachable — the only call site pinsagent.StreamingModeNone(harness.go:724, verified) and registry.go:379-380 documents the same caveat for cacheWriteMiddleware — but a one-line guard (pass pairs through unchanged whenstream == true) keeps this wrapper streaming-safe for when that path appears.@ -95,6 +97,7 @@ func NewRegistry(ctx context.Context, f *config.File) (*Registry, error) {}llm = &retryEmptyOutput{llm: llm, delay: emptyOutputRetryDelay}llm = &nudgeToolUse{llm: llm, delay: toolUseNudgeDelay}🟠 IMPORTANT — the two wrappers' budgets compose multiplicatively.
nudgeToolUsewrapsretryEmptyOutput, and each nudge round (up totoolUseNudgeMax= 2, so 3 calls total) callsretryEmptyOutput.GenerateContentfresh, re-arming its own budget:maxEmptyOutputRetries = 30attempts /emptyOutputRetryBudget = 15 * time.Secondper call (retry.go, verified atc52065e). Worst case per flow model call: 3 × 31 = 93 full-conversation re-sends and ~45s — each re-send costs the whole conversation (~521k tokens/turn per the PR description). The condition is not exotic: retry.go documents the target model (melious/deepseek-v4-flash-0731) answering empty ~2/3 of the time, and this PR exists because the same model answers prose-only.go build ./...passes; the wiring isllm = &retryEmptyOutput{...}; llm = &nudgeToolUse{llm: llm, ...}(lines 99-100). Fix: share one budget across both wrappers — pass a shared attempt counter / time budget into the chain, or count nudge re-sends against the same 15s/30-attempt budget instead of each round restarting it.Two independent causes produced the empty-output errors and the retry storms behind them ("adk run: openai: response output did not contain text or tool content"). 1. The <brainless-clock> block was appended to every outgoing request as a role:"system" item inside input, after the user message. The Responses API only sanctions system text as the top-level instructions field or a leading input item; melious's upstream silently drops a mid-conversation system item and answers HTTP 200 with nothing usable (its own error: "unsupported request parameter that the provider silently dropped"). Every round then hit the degenerate path. A probe isolating the shapes confirmed it: instructions-only 4/4 usable, clock-only 0/4, plain 2/2. The block is now a user-role content; it stays request-only (never persisted, never visible to the client), so the user never sees it. 2. The runner called the provider non-streaming (StreamingModeNone), the path melious implements worst: tool-bearing non-streaming requests came back with a reasoning item and no message 6/6 times in a probe, while streaming Responses returned usable turns 4/4 (pi drives melious the same way). The harness now runs StreamingModeSSE and processes only the final aggregated event, so the turn behaves exactly like a blocking call. The cache-write middleware now parses the SSE body's terminal usage event, so write billing survives streaming. Defense in depth stays in place, retuned: - retryEmptyOutput also retries ErrResponseFailed, and when its budget on the Responses endpoint exhausts it tries a Chat Completions client for the same provider/model once (both share the cache-write box). - Exhaustion now surfaces a human-readable error ("model X returned no usable output N times in a row...") instead of the raw ADK error, with the original kept in the log. - A per-provider api config option ("responses" default | "chat_completions") lets an operator pin a provider to Chat Completions. - The nudge (text-only stops without tool work) now judges the final aggregated response of a stream, never an opening delta. Also fixes the long-standing varnamelen on NewRegistry, so golangci-lint is clean.The memory tools summarized their work as counts ("Created 2 entities, 1 relation."), so a client rendering the tool-call result showed nothing about the actual memory content. Two changes make it visible: - The tool results now carry the content: create_memory names each created entity (with its type and quoted observations) and each relation triple; add_observations names the entity and quotes the added observations; delete_memory names the deleted entities. - The completed tool-call update for memory tools (create_memory, add_observations, delete_memory, read_graph, load_memory, preload_memory) now also carries the result text as a standard content block, the ACP field clients render in the timeline, alongside rawOutput. Tests: unit test on the enriched tool outputs; e2e test proving a create_memory result reaches the client as a rendered text content block.Review of PR #32 (fix/nudge-prose-only-turns), verified against head
0740791in a fresh worktree (Go 1.27.1, ADK v2.5.0).What holds up:
go vet ./...clean;go test ./...andgo test -race ./internal/harness/green at head; the PR's own tests pass.session.Eventembedsmodel.LLMResponsesoev.Partialexists (session/session.go:100-101, model/llm.go:56);openaimodel.ErrResponseFailed/ErrNoTextOrToolContent/ErrNoOutputItems/ErrNoChoicesexist (model/openaimodel/errors.go);ClientConfig.API+APIChatCompletionsexist (model/openaimodel/openaimodel.go:31-41). No go.mod/go.sum changes.One BLOCKING finding: the new nudge guard
hasToolWorkThisTurn(nudge.go:198) is defeated by the turn-clock block, which this very PR turns into a user-role text content appended to every request (harness.go:315-323, run before the model call per ADK v2.5.0 internal/llminternal/base_flow.go:957-966). Reproduced with a temp e2e test against the real harness: a turn that should complete in 2 model calls makes 4, with twonudging it to continueWARNs fired on the final answer. Every text-only final answer after tool work is discarded and re-sent with the nudge (up to 2×) — extra full-conversation re-sends, extra cost, and the model is told to call tools after the task is done. Evidence is in the inline comment; the PR's e2e stub serves the same text for every request ≥ 2, so its test cannot catch this.Verdict: REQUEST_CHANGES (1 blocking + 5 suggestions).
@ -661,0 +810,4 @@// The discarded first prose is never streamed; only the nudged round's// narrative and the final answer reach the client.wantUpdates := []string{"Let me look at the file.", "Found it: pong."}🟡 SUGGESTION — the stub's
defaultbranch of newProseThenToolResponsesServer serves the same final text for every request ≥ 2, so this test passes even when the nudge misfires on the final answer: with the guard broken the harness still produces the sameclient.updates(4 model calls; the discarded answers are never streamed). That is exactly what my failing e2e run produced. Add an assertion on the total request count (== 3) so a guard regression fails this test loudly.@ -1197,1 +1206,4 @@// unknown call into "other".// isMemoryTool reports whether name is one of the memory curation/search// tools, whose results the client should render as text.func (h *Harness) isMemoryTool(name string) bool {🟡 SUGGESTION —
isMemoryToolis inserted betweentoolKind's doc comment (lines 1199-1206) and thetoolKindfunction (line 1217), so the doc comment now documentsisMemoryTool— which it does not describe — andtoolKindis left undocumented. MoveisMemoryToolbelowtoolKind.@ -0,0 +195,4 @@// exactly this turn's activity: a tool round appends the model's function// calls and the tool results, so a text-only response after them is a final// answer, not a premature stop.func hasToolWorkThisTurn(contents []*genai.Content) bool {🔴 BLOCKING —
hasToolWorkThisTurnis defeated by the turn-clock block, so the nudge misfires on legitimate final answers after tool work.Reproduced against the PR head with temp tests in the harness package:
go test ./internal/harness/ -run TestReviewClockBlockDefeatsToolWorkGuard→ FAIL:hasToolWorkThisTurn = falsefor a conversation containing a function call + function response, because the appended user-role clock block (<brainless-clock>...) is the last user-role TEXT content and the tail after it is empty.go test ./internal/harness/ -run TestReviewFinalAnswerAfterToolWorkNotNudged→ FAIL:model called 4 times, want 2with twoWARN model answered with text and no tool call; nudging it to continuelines fired on the final answer.Why: harness.go:315-323 appends the clock block as a user-role text content to every outgoing request (ADK v2.5.0 internal/llminternal/base_flow.go:957-966 runs BeforeModelCallbacks before
m.GenerateContent), so this loop always lands on the clock block as the "last user text message" andcontents[lastUserText+1:]is empty. The guard!hasToolWorkThisTurn(req.Contents)in eligibleForToolUseNudge (line 144) is therefore always true in production; TestNudgeToolUseSkipsWhenToolsRanThisTurn passes only because its fixture has no clock block.Impact: every turn ending with a text-only final answer after tool work gets that answer discarded and re-sent with the nudge up to 2× — two extra full-conversation model calls per turn (latency + prefill cost) — and the model is instructed to call tools after the task is done. This is exactly the case the guard's own doc comment says must pass through untouched.
Fix: skip the clock block in the scan (ignore user-role contents whose text starts with
<brainless-clock>), or anchor the "last user text" scan before the trailing clock block.🟡 SUGGESTION — with StreamingModeSSE now always on (harness.go:724), this middleware's
io.ReadAllbuffers the entire SSE stream (all deltas + the terminal event) before the client decodes anything: every Responses response is fully materialized in memory and the streaming client gains no incremental delivery at the HTTP layer. Fine for typical answers; worth a cap or a streaming read if long streams become common.@ -126,0 +160,4 @@slog.String("model", r.llm.Name()),slog.Int("attempts", attempts))fb := (&retryEmptyOutput{llm: r.fallback, delay: r.delay}).GenerateContent(ctx, req, stream)🟡 SUGGESTION — the fallback is double-wrapped: NewRegistry already wraps the chat-completions client in
retryEmptyOutput(registry.go:134), and this line wraps it again, re-arming a full 15s/30-attempt retry budget on top of the exhausted primary budget — worst case ~30s of degenerate re-sends per model call before the human-readable failure. Callr.fallback.GenerateContent(ctx, req, stream)directly; the registry wrapper already provides the retry.Also,
fbPairs[0].err != nilonly inspects the first pair: a streamed fallback that yields deltas and then fails surfaces the raw ADK error instead of the human-readable message. Check the final pair.@ -114,3 +114,3 @@if got := req1.Contents[2].Role; got != "system" {if got := req1.Contents[2].Role; got != "user" {t.Fatalf("block role = %q, want system", got)🟡 SUGGESTION — the assertion now expects role "user" but the failure message still says
want system; update the message.Resolution record for reviews 58 (
c52065e) and 59 (0740791) — fixed at head6e89597,go test -race ./...+go vet ./...green,golangci-lint run ./...0 issues.🔴 1087 (BLOCKING — clock block defeats the tool-work guard): fixed.
hasToolWorkThisTurnnow skips user-role contents whose text starts with<brainless-clock>when locating the turn's last user message, so the guard sees the real conversation tail again. Regression tests:TestNudgeToolUseSkipsWhenClockBlockTrailsToolWork(tool round + trailing clock block passes through untouched) and the e2e now asserts exactly 3 model calls.🟡 1088 / 1081 (e2e cannot detect a wrong nudge on the final answer): fixed.
newProseThenToolResponsesServerexposes a request counter;TestACPNudgeKeepsTurnAliveAcrossProseOnlyAnswerassertsrequests() == 3, so a guard regression that nudges the final answer fails loudly.🟡 1089 (isMemoryTool stole toolKind's doc comment): fixed.
isMemoryToolmoved belowtoolKind;toolKind's doc comment documentstoolKindagain.🟡 1090 (stale 'want system' message): fixed. The turn-clock test now expects role "user" in its failure message.
🟡 1091 (fallback double-wrapped + first-pair check): fixed.
fallbackOrGiveUpcallsr.fallback.GenerateContentdirectly (the registry already wraps the chat-completions client in its own retry), the failure check reads the final pair so a streamed fallback that fails surfaces the human-readable message, and the fallback gets a fresh budget (the shared one is exhausted).🟡 1092 (unbounded middleware buffering): fixed. The cache-write middleware reads the body under a 64 MiB cap (
maxResponseBodyBytes) and fails the call clearly over it instead of holding unbounded memory.🟠 1079 (retry × nudge budgets compose multiplicatively): fixed. The retry and nudge wrappers now share one per-call degenerate-attempt budget carried in the context the nudge installs (
retryBudget/budgetFor/spend), so the worst case is one 30-attempt/15s cap instead of 3 × 31 = 93 re-sends.TestNudgeAndRetryShareOneBudgetpins the bound: an alternating prose/degenerate provider stops after 33 attempts (was 43+ unshared).🟡 1080 (stream flag ignored): resolved by the streaming work in
5f423e4— the harness now runs StreamingModeSSE, and the nudge decides on the final non-partial response (finalResponse), so partial deltas pass through and never trigger a nudge; both wrappers passstreamthrough unchanged.