feat: cost awareness for brainless agents (tiers 0-3) #25
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
sam/brainless!25
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/cost-awareness"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements cost_plan.md in full: tiers 0-3 (milestones C0-C5).
pricingblock (rates per million tokens, currency label; absence means free), a pureusageCostfunction (cached tokens subtracted, reasoning counted inside candidates, genai-native remainder added), a per-round cost line in the<brainless-clock>block, turn-log cost fields, compaction spend tracked via acostingLLMdecorator (the runner never yields the summary event), and once-per-process session-cost seeding from persisted events.budgets.maxTaskCost, a string so0.50cannot be misread as tokens); warns at the timeout's 50%/90% fractions and cancels the turn asStopReasonCancelledthrough the existing context path. A money budget with no priced model is a boot error.history.lastTasks):history: last 3 tasks averaged EUR 0.21 and 55k tokens, averaged from persisted events grouped by InvocationID, skipped when there are not enough completed tasks.Design deviations from the reviewed plan, per review: absence of pricing means free (no
free:trueflag, no boot warning), budgets are money-only per task (no token or session budgets), currency default is EUR.Verified: go build, go vet, go test ./... (all packages), golangci-lint run (0 issues).
AI review (samber/cc-skills-golang skill set) of cost awareness tiers 0-3.
Verified on head
b5187c6(go1.27.1):go build ./...clean,go vet ./...clean,go test ./...green for all packages. No dependency changes (go.mod/go.sum untouched). The cost arithmetic (usageCost), the free-by-absence design, the compaction decorator, and the seed-once guard are well thought out and well tested.Two 🟠 IMPORTANT findings, both in the Tier 2 budget machinery:
parseBudgetMoneyaccepts NaN and +Inf: NaN silently disables the budget with no boot error, +Inf makes a budget that can never fire — contradicting the documented "not a non-negative number is a boot error" rule.Three 🟡 suggestions (history line counts interrupted invocations as completed; missing negative-output-rate validation case; wrong expected-value literals in TestTurnClockCompactionBucket failure messages).
Recommendation: address the two IMPORTANT findings before merge, or explicitly accept the cancelled-reclassification semantics with a documented rationale and client-side handling. No blocking (🔴) issues found.
@ -306,0 +322,4 @@{name: "no pricing block is free", cfg: modelJSON(`{}`)},{name: "currency label", cfg: modelJSON(`{"pricing":{"currency":"EUR","inputPerMillion":0.27,"outputPerMillion":1.1}}`)},{name: "negative input rate",🟡 SUGGESTION — ValidatePricing checks all three rates (config.go:
InputPerMillion < 0 || CachedInputPerMillion < 0 || OutputPerMillion < 0) but the table only exercises negative input and negative cached rates. Add a "negative output rate" case so the third branch is pinned.@ -665,0 +754,4 @@// The in-loop budget check can lag one event behind a lazy usage fold, so// re-check once the runner is fully done: the final round's crossing must// still cancel the turn.if h.clock != nil && h.clock.budgetExceeded() {🟠 IMPORTANT — a completed turn is reclassified as cancelled when the final round (or post-turn compaction) crosses the budget. The usage fold in selectorModel.GenerateContent runs only after the runner has consumed the response (the wrapper folds post-range; ADK base_flow.go:772-803 yields the model event before the range ends), so the in-loop check at line 706 never sees the crossing round's cost, and this re-check cancels an already-finished turn.
Reproduced: same setup as TestACPCostBudgetSettlesTurn (mock returns a completed "pong", no tool calls). Without a budget the stop reason is EndTurn and the client receives the answer; with the budget crossed by that single round, the answer is still fully streamed to the client but the stop reason is "cancelled" (demo test logged:
answer was fully streamed to the client, stop reason = "cancelled"; the PR's own test passes asserting cancelled for this completed turn).Condition: the round that delivers the final answer pushes taskCost over budgetTaskCost — likely the most common crossing shape, since the model is told to wrap up at 90%.
Consequence: the ACP client (Paseo) sees a fully answered turn marked cancelled and may re-run the prompt, doubling the spend the budget exists to prevent. The turn log says "prompt complete" while the client sees cancelled — inconsistent signals.
Fix direction: fold usage when the response arrives (inside the yield loop) so the in-loop check can cancel before the final answer is streamed, and only report StopReasonCancelled when the turn actually aborted — a turn whose answer was delivered should stay EndTurn. At minimum, document and accept the semantics.
@ -1286,0 +1551,4 @@// parseBudgetMoney parses one money budget. Empty means off; a value that is// not a non-negative number is a boot error.func parseBudgetMoney(name, value string) (float64, error) {🟠 IMPORTANT — strconv.ParseFloat accepts NaN and ±Inf, and both silently defeat the money budget. Verified:
go runshows ParseFloat("NaN") → NaN, err=nil;NaN < 0is false so the negative check passes;budgetTaskCost > 0is then false in New, so a "NaN" budget is silently off with no boot error. "+Inf" parses clean, passes the > 0 priced-model check, buttaskCost >= +Infis never true, so the budget can never fire. The doc comment promises "a value that is not a non-negative number is a boot error" — NaN/Inf are not non-negative numbers yet pass. Fix: reject non-finite values, e.g.if math.IsNaN(money) || math.IsInf(money, 0) { return 0, fmt.Errorf("%s must be a finite non-negative number", name) }.@ -1286,0 +1512,4 @@}}if len(groups) < n {🟡 SUGGESTION — the history line is documented as "last N completed tasks", but any invocation with usage counts as completed, including one interrupted by timeout, preemption, the money budget, or a restart mid-turn (its prior rounds' events are persisted). So a cancelled turn still inflates the averaged history (and on a fresh-process resume, the invocation cut off by the restart counts too). Display-only inaccuracy; either filter for a completion marker (e.g. the invocation's last event being a model turn followed by a user message) or soften the label to "tasks started".
@ -396,0 +626,4 @@}}func TestTurnClockCompactionBucket(t *testing.T) {🟡 SUGGESTION — the failure messages in this new test quote the wrong expected values: "want 10500 / 0.011" (assertion wants 0.0215), "want 0.011 / 0.011" (asserts 0.0215), "want 0.011 / 0" (asserts 0.043). When the test fails, the message misleads; update the literals to match the assertions.
All five findings from the AI review resolved in
62812f2(pushed).🟠 IMPORTANT 1 — completed turn reclassified as cancelled: fixed per the suggested direction.
selectorModelnow folds usage inside its yield loop (before the response is handed to ADK), so the in-loop budget check sees the crossing round's cost and cancels before that event is streamed; the crossing event is skipped entirely. The end-of-loop re-check is removed, so a turn whose answer was delivered staysEndTurn.TestACPCostBudgetSettlesTurnnow also asserts the client received zero text updates on a budget-crossed turn — no complete answer marked cancelled, no re-run incentive.🟠 IMPORTANT 2 — NaN/Inf money budgets:
parseBudgetMoneynow rejects non-finite values (must be a finite non-negative number), soNaNcannot silently disable the budget and+Infcannot make an unfirable one. Test cases added for both.🟡 3 — history counts interrupted invocations:
taskHistorynow drops any invocation whose final event did not deliver an answer (pending tool call or no prose), so timeout/preemption/budget/restart-interrupted tasks no longer inflate the average.TestTaskHistorygained an interrupted-invocation case.🟡 4 — negative output rate case: added to
TestPricingValidation.🟡 5 — stale expected-value literals:
TestTurnClockCompactionBucketfailure messages now quote the asserted values (0.0215 / 0.0215 / 0.043).Verified:
go build,go vet,go test ./...green,golangci-lint run0 issues.