feat: cost awareness for brainless agents (tiers 0-3) #25

Merged
sam merged 12 commits from feat/cost-awareness into main 2026-10-05 17:35:57 +00:00
Collaborator

Implements cost_plan.md in full: tiers 0-3 (milestones C0-C5).

  • Tier 0 - the model is told tokens cost money, with the guardrail against corner-cutting (a wrong answer costs more than the tokens it saved).
  • Tier 1 - optional per-model pricing block (rates per million tokens, currency label; absence means free), a pure usageCost function (cached tokens subtracted, reasoning counted inside candidates, genai-native remainder added), a per-round cost line in the <brainless-clock> block, turn-log cost fields, compaction spend tracked via a costingLLM decorator (the runner never yields the summary event), and once-per-process session-cost seeding from persisted events.
  • Tier 2 - opt-in per-task money budget (budgets.maxTaskCost, a string so 0.50 cannot be misread as tokens); warns at the timeout's 50%/90% fractions and cancels the turn as StopReasonCancelled through the existing context path. A money budget with no priced model is a boot error.
  • Tier 3 - measured-history line on priced models (history.lastTasks): history: last 3 tasks averaged EUR 0.21 and 55k tokens, averaged from persisted events grouped by InvocationID, skipped when there are not enough completed tasks.

Design deviations from the reviewed plan, per review: absence of pricing means free (no free:true flag, no boot warning), budgets are money-only per task (no token or session budgets), currency default is EUR.

Verified: go build, go vet, go test ./... (all packages), golangci-lint run (0 issues).

Implements cost_plan.md in full: tiers 0-3 (milestones C0-C5). - **Tier 0** - the model is told tokens cost money, with the guardrail against corner-cutting (a wrong answer costs more than the tokens it saved). - **Tier 1** - optional per-model `pricing` block (rates per million tokens, currency label; absence means free), a pure `usageCost` function (cached tokens subtracted, reasoning counted inside candidates, genai-native remainder added), a per-round cost line in the `<brainless-clock>` block, turn-log cost fields, compaction spend tracked via a `costingLLM` decorator (the runner never yields the summary event), and once-per-process session-cost seeding from persisted events. - **Tier 2** - opt-in per-task money budget (`budgets.maxTaskCost`, a string so `0.50` cannot be misread as tokens); warns at the timeout's 50%/90% fractions and cancels the turn as `StopReasonCancelled` through the existing context path. A money budget with no priced model is a boot error. - **Tier 3** - measured-history line on priced models (`history.lastTasks`): `history: last 3 tasks averaged EUR 0.21 and 55k tokens`, averaged from persisted events grouped by InvocationID, skipped when there are not enough completed tasks. Design deviations from the reviewed plan, per review: absence of pricing means free (no `free:true` flag, no boot warning), budgets are money-only per task (no token or session budgets), currency default is EUR. Verified: go build, go vet, go test ./... (all packages), golangci-lint run (0 issues).
Tier 1 of cost_plan.md on top of the tier-0 framing sentence:
- brainless.json gains an optional per-model pricing block (rates per
  million tokens, free:true, currency label); validation in config.Validate
  and mirrored in harness.New, boot warning for unpriced models.
- turnClock accumulates output/thought/compaction tokens plus last-round,
  task and session cost from a pure usageCost function (cached tokens
  subtracted, reasoning counted inside candidates, genai-native remainder
  added).
- the <brainless-clock> block renders a cost line on priced models only;
  the turn log gains output_tokens, thought_tokens, cost, session_cost and
  compaction spend.
- compaction spend is tracked through a costingLLM decorator (the runner
  never yields the summary event), kept out of the prompt figures.
- a resumed session seeds sessionCost once from persisted event usage;
  the lastPromptEnded guard prevents double-counting across restarts.
- Pricing struct uses omitzero tags, matching the repo convention
  (config.go Timeouts/Compaction).
- The turn log only allocates cost attrs when sessionCost > 0; a free or
  unpriced model no longer logs cost=0 forever.
- seedFromPersistedSession drops the never-nil logger guard; the store-read
  error keeps its Debug sink (errhandle requires it) without the guard.
- costingLLM forwards Name/GenerateContent explicitly instead of embedding
  model.LLM, so a new interface method fails to compile here rather than
  silently passing through.
Per review: no cost set = free. Removes the Free field, the
input/output-rate requirement, the boot warning for unpriced models and
the three-state design; validation is now only that rates are not
negative. Pricing.IsFree() covers both nil and a block with no rates.
cost_plan.md updated to match.
Tier 2 of cost_plan.md, off by default:
- config gains a sibling budgets block: maxTaskTokens (works on every
  model), maxTaskCost and maxSessionCost (strings, so 0.50 cannot be
  misread as tokens). A money budget with no priced model anywhere is a
  boot error rather than a silent no-op.
- the clock block warns at the timeout's fractions: informational at 50%,
  a 'report what you have' directive at 90%, keyed off tokens and money.
- a crossing cancels the turn through the existing context path and
  settles as StopReasonCancelled; the in-loop check stops multi-round
  tasks and an end-of-loop re-check catches the final round, whose usage
  fold can lag one event behind under lazy iteration.
Per review: steer on time and money, not tokens; no session budget (a
session is not something you stop). budgets.maxTaskCost is the only
budget: maxTaskTokens and maxSessionCost are gone, the clock warns on the
task cost alone, and budgetExceeded checks taskCost per prompt. The turn
cancel func is renamed to cancelTurn for clarity. cost_plan.md Tier 2
updated to match.
feat: show measured task history to the model (tier 3)
All checks were successful
code-review / Agent review (pull_request) Successful in 8s
b5187c6273
Tier 3 of cost_plan.md: the clock block gains a measured-history line on
priced models, e.g. 'history: last 3 tasks averaged EUR 0.21 and 55k
tokens'. It is an average of completed tasks, never a prediction:
- config.history.lastTasks enables it; zero/absent means off.
- taskHistory reads the session once per prompt (only when enabled), groups
  stored events by InvocationID, and takes the last n completed invocations
  by completion time, priced at the active model since events do not carry
  the model that produced them.
- the line is skipped when fewer than n invocations exist (no extrapolating
  from one) or when no usage was reported; start clears the fields so a
  failed read never leaves stale history on the block.
pi-reviewer left a comment

AI review (samber/cc-skills-golang skill set) of cost awareness tiers 0-3.

Verified on head b5187c6 (go1.27.1): go build ./... clean, go vet ./... clean, go test ./... green for all packages. No dependency changes (go.mod/go.sum untouched). The cost arithmetic (usageCost), the free-by-absence design, the compaction decorator, and the seed-once guard are well thought out and well tested.

Two 🟠 IMPORTANT findings, both in the Tier 2 budget machinery:

  1. A turn that fully completes is reclassified as StopReasonCancelled when the final round (or post-turn compaction) crosses the budget — after the answer was already fully streamed to the client. The client sees a complete answer marked cancelled and may re-run, doubling the spend the budget exists to prevent. Reproduced against the PR's own e2e harness; the PR's TestACPCostBudgetSettlesTurn pins exactly this behavior (mock returns a completed response, no tool calls).
  2. parseBudgetMoney accepts NaN and +Inf: NaN silently disables the budget with no boot error, +Inf makes a budget that can never fire — contradicting the documented "not a non-negative number is a boot error" rule.

Three 🟡 suggestions (history line counts interrupted invocations as completed; missing negative-output-rate validation case; wrong expected-value literals in TestTurnClockCompactionBucket failure messages).

Recommendation: address the two IMPORTANT findings before merge, or explicitly accept the cancelled-reclassification semantics with a documented rationale and client-side handling. No blocking (🔴) issues found.

AI review (samber/cc-skills-golang skill set) of cost awareness tiers 0-3. Verified on head b5187c6 (go1.27.1): `go build ./...` clean, `go vet ./...` clean, `go test ./...` green for all packages. No dependency changes (go.mod/go.sum untouched). The cost arithmetic (`usageCost`), the free-by-absence design, the compaction decorator, and the seed-once guard are well thought out and well tested. Two 🟠 IMPORTANT findings, both in the Tier 2 budget machinery: 1. A turn that fully completes is reclassified as StopReasonCancelled when the final round (or post-turn compaction) crosses the budget — after the answer was already fully streamed to the client. The client sees a complete answer marked cancelled and may re-run, doubling the spend the budget exists to prevent. Reproduced against the PR's own e2e harness; the PR's TestACPCostBudgetSettlesTurn pins exactly this behavior (mock returns a completed response, no tool calls). 2. `parseBudgetMoney` accepts NaN and +Inf: NaN silently disables the budget with no boot error, +Inf makes a budget that can never fire — contradicting the documented "not a non-negative number is a boot error" rule. Three 🟡 suggestions (history line counts interrupted invocations as completed; missing negative-output-rate validation case; wrong expected-value literals in TestTurnClockCompactionBucket failure messages). Recommendation: address the two IMPORTANT findings before merge, or explicitly accept the cancelled-reclassification semantics with a documented rationale and client-side handling. No blocking (🔴) issues found.
@ -306,0 +322,4 @@
{name: "no pricing block is free", cfg: modelJSON(`{}`)},
{name: "currency label", cfg: modelJSON(`{"pricing":{"currency":"EUR","inputPerMillion":0.27,"outputPerMillion":1.1}}`)},
{
name: "negative input rate",
Collaborator

🟡 SUGGESTION — ValidatePricing checks all three rates (config.go: InputPerMillion < 0 || CachedInputPerMillion < 0 || OutputPerMillion < 0) but the table only exercises negative input and negative cached rates. Add a "negative output rate" case so the third branch is pinned.

🟡 SUGGESTION — ValidatePricing checks all three rates (config.go: `InputPerMillion < 0 || CachedInputPerMillion < 0 || OutputPerMillion < 0`) but the table only exercises negative input and negative cached rates. Add a "negative output rate" case so the third branch is pinned.
@ -665,0 +754,4 @@
// The in-loop budget check can lag one event behind a lazy usage fold, so
// re-check once the runner is fully done: the final round's crossing must
// still cancel the turn.
if h.clock != nil && h.clock.budgetExceeded() {
Collaborator

🟠 IMPORTANT — a completed turn is reclassified as cancelled when the final round (or post-turn compaction) crosses the budget. The usage fold in selectorModel.GenerateContent runs only after the runner has consumed the response (the wrapper folds post-range; ADK base_flow.go:772-803 yields the model event before the range ends), so the in-loop check at line 706 never sees the crossing round's cost, and this re-check cancels an already-finished turn.

Reproduced: same setup as TestACPCostBudgetSettlesTurn (mock returns a completed "pong", no tool calls). Without a budget the stop reason is EndTurn and the client receives the answer; with the budget crossed by that single round, the answer is still fully streamed to the client but the stop reason is "cancelled" (demo test logged: answer was fully streamed to the client, stop reason = "cancelled"; the PR's own test passes asserting cancelled for this completed turn).

Condition: the round that delivers the final answer pushes taskCost over budgetTaskCost — likely the most common crossing shape, since the model is told to wrap up at 90%.

Consequence: the ACP client (Paseo) sees a fully answered turn marked cancelled and may re-run the prompt, doubling the spend the budget exists to prevent. The turn log says "prompt complete" while the client sees cancelled — inconsistent signals.

Fix direction: fold usage when the response arrives (inside the yield loop) so the in-loop check can cancel before the final answer is streamed, and only report StopReasonCancelled when the turn actually aborted — a turn whose answer was delivered should stay EndTurn. At minimum, document and accept the semantics.

🟠 IMPORTANT — a completed turn is reclassified as cancelled when the final round (or post-turn compaction) crosses the budget. The usage fold in selectorModel.GenerateContent runs only after the runner has consumed the response (the wrapper folds post-range; ADK base_flow.go:772-803 yields the model event before the range ends), so the in-loop check at line 706 never sees the crossing round's cost, and this re-check cancels an already-finished turn. Reproduced: same setup as TestACPCostBudgetSettlesTurn (mock returns a completed "pong", no tool calls). Without a budget the stop reason is EndTurn and the client receives the answer; with the budget crossed by that single round, the answer is still fully streamed to the client but the stop reason is "cancelled" (demo test logged: `answer was fully streamed to the client, stop reason = "cancelled"`; the PR's own test passes asserting cancelled for this completed turn). Condition: the round that delivers the final answer pushes taskCost over budgetTaskCost — likely the most common crossing shape, since the model is told to wrap up at 90%. Consequence: the ACP client (Paseo) sees a fully answered turn marked cancelled and may re-run the prompt, doubling the spend the budget exists to prevent. The turn log says "prompt complete" while the client sees cancelled — inconsistent signals. Fix direction: fold usage when the response arrives (inside the yield loop) so the in-loop check can cancel before the final answer is streamed, and only report StopReasonCancelled when the turn actually aborted — a turn whose answer was delivered should stay EndTurn. At minimum, document and accept the semantics.
@ -1286,0 +1551,4 @@
// parseBudgetMoney parses one money budget. Empty means off; a value that is
// not a non-negative number is a boot error.
func parseBudgetMoney(name, value string) (float64, error) {
Collaborator

🟠 IMPORTANT — strconv.ParseFloat accepts NaN and ±Inf, and both silently defeat the money budget. Verified: go run shows ParseFloat("NaN") → NaN, err=nil; NaN < 0 is false so the negative check passes; budgetTaskCost > 0 is then false in New, so a "NaN" budget is silently off with no boot error. "+Inf" parses clean, passes the > 0 priced-model check, but taskCost >= +Inf is never true, so the budget can never fire. The doc comment promises "a value that is not a non-negative number is a boot error" — NaN/Inf are not non-negative numbers yet pass. Fix: reject non-finite values, e.g. if math.IsNaN(money) || math.IsInf(money, 0) { return 0, fmt.Errorf("%s must be a finite non-negative number", name) }.

🟠 IMPORTANT — strconv.ParseFloat accepts NaN and ±Inf, and both silently defeat the money budget. Verified: `go run` shows ParseFloat("NaN") → NaN, err=nil; `NaN < 0` is false so the negative check passes; `budgetTaskCost > 0` is then false in New, so a "NaN" budget is silently off with no boot error. "+Inf" parses clean, passes the > 0 priced-model check, but `taskCost >= +Inf` is never true, so the budget can never fire. The doc comment promises "a value that is not a non-negative number is a boot error" — NaN/Inf are not non-negative numbers yet pass. Fix: reject non-finite values, e.g. `if math.IsNaN(money) || math.IsInf(money, 0) { return 0, fmt.Errorf("%s must be a finite non-negative number", name) }`.
@ -1286,0 +1512,4 @@
}
}
if len(groups) < n {
Collaborator

🟡 SUGGESTION — the history line is documented as "last N completed tasks", but any invocation with usage counts as completed, including one interrupted by timeout, preemption, the money budget, or a restart mid-turn (its prior rounds' events are persisted). So a cancelled turn still inflates the averaged history (and on a fresh-process resume, the invocation cut off by the restart counts too). Display-only inaccuracy; either filter for a completion marker (e.g. the invocation's last event being a model turn followed by a user message) or soften the label to "tasks started".

🟡 SUGGESTION — the history line is documented as "last N completed tasks", but any invocation with usage counts as completed, including one interrupted by timeout, preemption, the money budget, or a restart mid-turn (its prior rounds' events are persisted). So a cancelled turn still inflates the averaged history (and on a fresh-process resume, the invocation cut off by the restart counts too). Display-only inaccuracy; either filter for a completion marker (e.g. the invocation's last event being a model turn followed by a user message) or soften the label to "tasks started".
@ -396,0 +626,4 @@
}
}
func TestTurnClockCompactionBucket(t *testing.T) {
Collaborator

🟡 SUGGESTION — the failure messages in this new test quote the wrong expected values: "want 10500 / 0.011" (assertion wants 0.0215), "want 0.011 / 0.011" (asserts 0.0215), "want 0.011 / 0" (asserts 0.043). When the test fails, the message misleads; update the literals to match the assertions.

🟡 SUGGESTION — the failure messages in this new test quote the wrong expected values: "want 10500 / 0.011" (assertion wants 0.0215), "want 0.011 / 0.011" (asserts 0.0215), "want 0.011 / 0" (asserts 0.043). When the test fails, the message misleads; update the literals to match the assertions.
- Budget crossing now folds usage inside the model wrapper's yield loop,
  so the in-loop check sees the crossing round and aborts before the
  answer is streamed; the end-of-loop re-check is gone, so a turn whose
  answer was delivered stays EndTurn. The e2e pins that a budget-crossed
  turn delivers no text to the client.
- parseBudgetMoney rejects NaN and +/-Inf (a 'NaN' budget was silently
  off, '+Inf' could never fire), with test cases.
- taskHistory only counts invocations whose final event delivered an
  answer; an interrupted task (timeout, preemption, budget, restart)
  ends with a pending tool call or no prose and is excluded, with an
  interrupted-invocation test case.
- TestPricingValidation covers a negative output rate.
- TestTurnClockCompactionBucket failure messages quote the asserted
  values.
Author
Collaborator

All five findings from the AI review resolved in 62812f2 (pushed).

🟠 IMPORTANT 1 — completed turn reclassified as cancelled: fixed per the suggested direction. selectorModel now folds usage inside its yield loop (before the response is handed to ADK), so the in-loop budget check sees the crossing round's cost and cancels before that event is streamed; the crossing event is skipped entirely. The end-of-loop re-check is removed, so a turn whose answer was delivered stays EndTurn. TestACPCostBudgetSettlesTurn now also asserts the client received zero text updates on a budget-crossed turn — no complete answer marked cancelled, no re-run incentive.

🟠 IMPORTANT 2 — NaN/Inf money budgets: parseBudgetMoney now rejects non-finite values (must be a finite non-negative number), so NaN cannot silently disable the budget and +Inf cannot make an unfirable one. Test cases added for both.

🟡 3 — history counts interrupted invocations: taskHistory now drops any invocation whose final event did not deliver an answer (pending tool call or no prose), so timeout/preemption/budget/restart-interrupted tasks no longer inflate the average. TestTaskHistory gained an interrupted-invocation case.

🟡 4 — negative output rate case: added to TestPricingValidation.

🟡 5 — stale expected-value literals: TestTurnClockCompactionBucket failure messages now quote the asserted values (0.0215 / 0.0215 / 0.043).

Verified: go build, go vet, go test ./... green, golangci-lint run 0 issues.

All five findings from the AI review resolved in 62812f2 (pushed). 🟠 **IMPORTANT 1 — completed turn reclassified as cancelled:** fixed per the suggested direction. `selectorModel` now folds usage inside its yield loop (before the response is handed to ADK), so the in-loop budget check sees the crossing round's cost and cancels *before* that event is streamed; the crossing event is skipped entirely. The end-of-loop re-check is removed, so a turn whose answer was delivered stays `EndTurn`. `TestACPCostBudgetSettlesTurn` now also asserts the client received **zero** text updates on a budget-crossed turn — no complete answer marked cancelled, no re-run incentive. 🟠 **IMPORTANT 2 — NaN/Inf money budgets:** `parseBudgetMoney` now rejects non-finite values (`must be a finite non-negative number`), so `NaN` cannot silently disable the budget and `+Inf` cannot make an unfirable one. Test cases added for both. 🟡 **3 — history counts interrupted invocations:** `taskHistory` now drops any invocation whose final event did not deliver an answer (pending tool call or no prose), so timeout/preemption/budget/restart-interrupted tasks no longer inflate the average. `TestTaskHistory` gained an interrupted-invocation case. 🟡 **4 — negative output rate case:** added to `TestPricingValidation`. 🟡 **5 — stale expected-value literals:** `TestTurnClockCompactionBucket` failure messages now quote the asserted values (0.0215 / 0.0215 / 0.043). Verified: `go build`, `go vet`, `go test ./...` green, `golangci-lint run` 0 issues.
sam merged commit 51fe8e9d93 into main 2026-10-05 17:35:57 +00:00
sam deleted branch feat/cost-awareness 2026-10-05 17:35:57 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sam/brainless!25
No description provided.