feat: give agents a concept of time (clock, tool costs, timeout, preemption) #23

Merged
sam merged 8 commits from feat/time-awareness into main 2026-10-04 13:52:06 +00:00
Collaborator

What

Gives agents in the brainless harness a working sense of the passage of time, implemented across the four tiers of time_plan.md plus the Milestone 0 cancellation investigation.

Tier 1 — live clock and usage telemetry. Every model request carries a trailing system-role <brainless-clock> block: UTC now, elapsed time in the current task, round number, time since the user's last message, and a per-turn prompt complete log with rounds, prompt/cached tokens, cache-hit ratio, model time, and turn time. Cache-safe: the block is appended at the tail only, and a prefix-stability test pins that earlier content is never modified.

Tier 2 — tool durations. Each tool call is timed from its FunctionCall to FunctionResponse event; the clock block reports tools in this task: X total; slowest: NAME Y, the finished tool's result is tagged [took 4m2s] in the outgoing request, and the turn summary includes tool time and slowest call.

Tier 3 — per-prompt timeout. "timeouts": { "promptTimeout": "10m" } in brainless.json (Go duration string, boot-validated, off unless configured). The clock block gains a countdown at 50% of the budget and a directive at 90%; at expiry the turn settles with the partial answer and stop reason cancelled, and the in-flight model call is aborted.

Tier 4 — mid-turn preemption. A prompt sent while another is running preempts it at the next safe boundary: immediately when no tool call is in flight, otherwise once the tool finishes. A tool call is never severed; immediate interruption remains session/cancel.

Milestone 0 — ADK cancellation is not a bug. The long-standing "ADK does not abort in-flight model calls" claim was an artifact of a test server that never read its request body (Go's http server only starts the disconnect-watching background read once the body hits EOF). Pinned by TestADKCancelAbortsInflightModelCall plus two bisection tests (plain net/http and bare openai-go). Tiers 3 and 4 rely on this.

Also on this branch: a pre-existing flake fix — in-memory memory stores now get a unique per-instance database name, so parallel cmd/brainless tests no longer race each other's CREATE TABLE (fix(tools): isolate per-instance in-memory memory stores).

Commits

  • a9318af docs: add time-awareness implementation plan
  • 8f6a08b feat(harness): add live clock and usage telemetry to prompt turns
  • 22fb438 fix(tools): isolate per-instance in-memory memory stores
  • 886f279 test(harness): prove session/cancel aborts in-flight model calls (M0)
  • b79de12 feat(harness): report tool durations to the model (Tier 2)
  • 8dc917b feat(harness): add per-prompt timeout with escalating warnings (Tier 3)
  • e35d749 feat(harness): preempt a running prompt at the next safe boundary (Tier 4)

Verification

go build, go vet, full go test ./... (repeated), racedetector on the touched packages, golangci-lint — all clean. New tests cover the clock block, prefix stability, tool timing/tagging, timeout parsing and escalation, prompt-timeout settling, and mid-turn preemption.

## What Gives agents in the brainless harness a working sense of the passage of time, implemented across the four tiers of `time_plan.md` plus the Milestone 0 cancellation investigation. **Tier 1 — live clock and usage telemetry.** Every model request carries a trailing system-role `<brainless-clock>` block: UTC now, elapsed time in the current task, round number, time since the user's last message, and a per-turn `prompt complete` log with rounds, prompt/cached tokens, cache-hit ratio, model time, and turn time. Cache-safe: the block is appended at the tail only, and a prefix-stability test pins that earlier content is never modified. **Tier 2 — tool durations.** Each tool call is timed from its FunctionCall to FunctionResponse event; the clock block reports `tools in this task: X total; slowest: NAME Y`, the finished tool's result is tagged `[took 4m2s]` in the outgoing request, and the turn summary includes tool time and slowest call. **Tier 3 — per-prompt timeout.** `"timeouts": { "promptTimeout": "10m" }` in brainless.json (Go duration string, boot-validated, off unless configured). The clock block gains a countdown at 50% of the budget and a directive at 90%; at expiry the turn settles with the partial answer and stop reason `cancelled`, and the in-flight model call is aborted. **Tier 4 — mid-turn preemption.** A prompt sent while another is running preempts it at the next safe boundary: immediately when no tool call is in flight, otherwise once the tool finishes. A tool call is never severed; immediate interruption remains `session/cancel`. **Milestone 0 — ADK cancellation is not a bug.** The long-standing "ADK does not abort in-flight model calls" claim was an artifact of a test server that never read its request body (Go's http server only starts the disconnect-watching background read once the body hits EOF). Pinned by `TestADKCancelAbortsInflightModelCall` plus two bisection tests (plain net/http and bare openai-go). Tiers 3 and 4 rely on this. **Also on this branch:** a pre-existing flake fix — in-memory memory stores now get a unique per-instance database name, so parallel `cmd/brainless` tests no longer race each other's `CREATE TABLE` (`fix(tools): isolate per-instance in-memory memory stores`). ## Commits - `a9318af` docs: add time-awareness implementation plan - `8f6a08b` feat(harness): add live clock and usage telemetry to prompt turns - `22fb438` fix(tools): isolate per-instance in-memory memory stores - `886f279` test(harness): prove session/cancel aborts in-flight model calls (M0) - `b79de12` feat(harness): report tool durations to the model (Tier 2) - `8dc917b` feat(harness): add per-prompt timeout with escalating warnings (Tier 3) - `e35d749` feat(harness): preempt a running prompt at the next safe boundary (Tier 4) ## Verification `go build`, `go vet`, full `go test ./...` (repeated), `racedetector` on the touched packages, `golangci-lint` — all clean. New tests cover the clock block, prefix stability, tool timing/tagging, timeout parsing and escalation, prompt-timeout settling, and mid-turn preemption.
pi-reviewer left a comment

Review — feat/time-awareness (PR #23)

Reviewed the full diff (README, config, harness, registry, turnclock, memory, time_plan.md, and all tests) as pi-reviewer. Everything in this review was verified in this session against the head commit e35d749 and the pinned ADK v2.5.0 / openai-go v3.71.1 module sources, not from memory.

Verified clean: go build ./..., go vet, full go test ./internal/harness/ ./internal/tools/ ./internal/config/ (incl. the new TestACPMidTurnPromptPreempts, TestACPPromptTimeoutSettlesTurn, TestADKCancelAbortsInflightModelCall, cancel_http_test.go pair, and the turnclock suite), racedetector test ./internal/harness/ (no DATA RACE warnings), golangci-lint run (0 issues), govulncheck ./... (exit 0). No go.mod/go.sum changes vs main. The ADK BeforeModelCallback contract ((nil, nil) → continue to the model call) is confirmed in agent/llmagent/llmagent.go:373-375, and the merged parallel-tool-response event in internal/llminternal/base_flow.go:1614 makes the boolean toolInFlight tracking correct for parallel tool calls.

Findings: 1 IMPORTANT, 1 SUGGESTION (no BLOCKING).

  1. 🟠 IMPORTANT — internal/harness/registry.go:329: the selectorModel.GenerateContent wrapper accumulates the compaction summarizer's usage into the running turn's clock whenever compaction.summarizerModel is unset (the default). Verified: compactionFor leaves out.Summarizer nil in that case (internal/harness/harness.go:1200-1235), ADK's runner then installs NewLLMSummarizer over the root agent's model — this selectorModel — (runner/runner.go:172-201), and the summarizer calls s.model.GenerateContent(...) directly (llm_summarizer.go:232), landing in this wrapper. When compaction fires (default threshold 20k tokens — routine for long sessions), prompt_tokens/cached_tokens/model_time/cache_hit_ratio in the new "prompt complete" log are silently inflated by the summarization call — the exact "hidden spend from summarization" the plan flags. Fix: when SummarizerModel == "", install compaction.NewLLMSummarizer over the raw h.reg.LLM(ref) (as compactionFor already does for the configured case) so the summarizer bypasses the clock wrapper.

  2. 🟡 SUGGESTION — internal/harness/harness.go:448: the preemption check runs exactly once, before the prompt knows it will be queued. A prompt arriving in the microsecond window between the running turn's turnMu.Lock() and registerTurn sees turnCancel == nil, no-ops, and then waits for the whole turn; a prompt queued behind an already-preempting prompt has its one-shot check spent and never preempts the turn that runs ahead of it. Both silently fall back to the pre-PR "wait for the whole turn" behavior. Fix: replace the boolean with a queued-prompt counter (increment at Prompt entry when a turn is running, decrement on registration) and have the running turn cancel at the next round boundary when the counter is non-zero.

Question (not a finding): the clock block is appended as a trailing system-role content; the ADK Responses converter emits it in content order (model/openaimodel/internal/responses/request.go:267), i.e. a system input item after user/tool items. OpenAI's Responses API requires system instructions first, so strict servers may reject or ignore it. The plan flags this as a "remaining risk" — please confirm the trailing system message was actually exercised against the real providers (Melious, local llama.cpp), since all e2e tests in this PR run against a fake server that accepts anything.

## Review — feat/time-awareness (PR #23) Reviewed the full diff (README, config, harness, registry, turnclock, memory, time_plan.md, and all tests) as `pi-reviewer`. Everything in this review was verified in this session against the head commit `e35d749` and the pinned ADK v2.5.0 / openai-go v3.71.1 module sources, not from memory. **Verified clean:** `go build ./...`, `go vet`, full `go test ./internal/harness/ ./internal/tools/ ./internal/config/` (incl. the new `TestACPMidTurnPromptPreempts`, `TestACPPromptTimeoutSettlesTurn`, `TestADKCancelAbortsInflightModelCall`, `cancel_http_test.go` pair, and the turnclock suite), `racedetector test ./internal/harness/` (no DATA RACE warnings), `golangci-lint run` (0 issues), `govulncheck ./...` (exit 0). No `go.mod`/`go.sum` changes vs `main`. The ADK `BeforeModelCallback` contract (`(nil, nil)` → continue to the model call) is confirmed in `agent/llmagent/llmagent.go:373-375`, and the merged parallel-tool-response event in `internal/llminternal/base_flow.go:1614` makes the boolean `toolInFlight` tracking correct for parallel tool calls. **Findings: 1 IMPORTANT, 1 SUGGESTION (no BLOCKING).** 1. 🟠 IMPORTANT — `internal/harness/registry.go:329`: the `selectorModel.GenerateContent` wrapper accumulates the **compaction summarizer's** usage into the running turn's clock whenever `compaction.summarizerModel` is unset (the default). Verified: `compactionFor` leaves `out.Summarizer` nil in that case (`internal/harness/harness.go:1200-1235`), ADK's runner then installs `NewLLMSummarizer` over the root agent's model — this `selectorModel` — (`runner/runner.go:172-201`), and the summarizer calls `s.model.GenerateContent(...)` directly (`llm_summarizer.go:232`), landing in this wrapper. When compaction fires (default threshold 20k tokens — routine for long sessions), `prompt_tokens`/`cached_tokens`/`model_time`/`cache_hit_ratio` in the new "prompt complete" log are silently inflated by the summarization call — the exact "hidden spend from summarization" the plan flags. Fix: when `SummarizerModel == ""`, install `compaction.NewLLMSummarizer` over the raw `h.reg.LLM(ref)` (as `compactionFor` already does for the configured case) so the summarizer bypasses the clock wrapper. 2. 🟡 SUGGESTION — `internal/harness/harness.go:448`: the preemption check runs exactly once, before the prompt knows it will be queued. A prompt arriving in the microsecond window between the running turn's `turnMu.Lock()` and `registerTurn` sees `turnCancel == nil`, no-ops, and then waits for the whole turn; a prompt queued behind an already-preempting prompt has its one-shot check spent and never preempts the turn that runs ahead of it. Both silently fall back to the pre-PR "wait for the whole turn" behavior. Fix: replace the boolean with a queued-prompt counter (increment at Prompt entry when a turn is running, decrement on registration) and have the running turn cancel at the next round boundary when the counter is non-zero. **Question (not a finding):** the clock block is appended as a trailing system-role content; the ADK Responses converter emits it in content order (`model/openaimodel/internal/responses/request.go:267`), i.e. a system input item after user/tool items. OpenAI's Responses API requires system instructions first, so strict servers may reject or ignore it. The plan flags this as a "remaining risk" — please confirm the trailing system message was actually exercised against the real providers (Melious, local llama.cpp), since all e2e tests in this PR run against a fake server that accepts anything.
@ -404,0 +445,4 @@
// safe boundary: now when no tool call is in flight, otherwise when the
// tool finishes. A tool call in flight is never severed; immediate
// interruption remains session/cancel.
h.requestPreemption()
Collaborator

🟡 SUGGESTION — The preemption check runs exactly once, before this prompt knows whether it will actually be queued.

requestPreemption() executes before turnMu.Lock() (line 453) and registerTurn (line 460). Two timings silently fall back to the pre-PR "wait for the whole turn" behavior: (a) a prompt arriving in the window between the running turn's turnMu.Lock() and registerTurn sees turnCancel == nil and no-ops, then blocks for the entire turn; (b) a prompt queued behind an already-preempting prompt has its one-shot check already spent and never preempts the turn that runs ahead of it. Both are benign (no crash — old behavior) but make the Tier 4 guarantee timing-dependent.

Fix: replace the preemptPending boolean with a queued-prompt counter — increment at Prompt entry whenever a turn is registered, decrement when a waiting prompt acquires the turn — and have the running turn cancel at the next round boundary when the counter is non-zero (still never while a tool is in flight).

🟡 SUGGESTION — The preemption check runs exactly once, before this prompt knows whether it will actually be queued. `requestPreemption()` executes before `turnMu.Lock()` (line 453) and `registerTurn` (line 460). Two timings silently fall back to the pre-PR "wait for the whole turn" behavior: (a) a prompt arriving in the window between the running turn's `turnMu.Lock()` and `registerTurn` sees `turnCancel == nil` and no-ops, then blocks for the entire turn; (b) a prompt queued behind an already-preempting prompt has its one-shot check already spent and never preempts the turn that runs ahead of it. Both are benign (no crash — old behavior) but make the Tier 4 guarantee timing-dependent. Fix: replace the `preemptPending` boolean with a queued-prompt counter — increment at Prompt entry whenever a turn is registered, decrement when a waiting prompt acquires the turn — and have the running turn cancel at the next round boundary when the counter is non-zero (still never while a tool is in flight).
@ -296,0 +326,4 @@
elapsed := time.Since(start)
if m.clock != nil {
m.clock.addUsage(usage, elapsed)
Collaborator

🟠 IMPORTANT — Summarizer usage is accumulated into the running turn's clock when compaction.summarizerModel is unset.

This wrapper wraps every GenerateContent flowing through the root selectorModel. With the default config (compaction on, 20k-token threshold, no summarizerModel), compactionFor (internal/harness/harness.go:1200-1235) leaves out.Summarizer nil, so ADK's runner installs its default LLMSummarizer over the root agent's model — this selectorModel — (runner/runner.go:172-201: m := llminternal.Reveal(llmAgent).Model; NewLLMSummarizer(LLMSummarizerConfig{Model: m})). The summarizer calls s.model.GenerateContent(...) directly (llm_summarizer.go:232), so m.clock.addUsage adds the summarizer's prompt/cached tokens and call duration to the current turn, and the new "prompt complete" log reports inflated prompt_tokens/cached_tokens/model_time and a wrong cache_hit_ratio whenever compaction fires mid-turn.

Verified this session: sed -n '172,201p' /home/paseo/go/pkg/mod/google.golang.org/adk/v2@v2.5.0/runner/runner.go and sed -n '232p' .../session/compaction/llm_summarizer.go (direct GenerateContent, no callbacks), plus grep -n 'out.Summarizer' internal/harness/harness.go (compactionFor sets it only when SummarizerModel != "").

Fix: when SummarizerModel == "", install compaction.NewLLMSummarizer(compaction.LLMSummarizerConfig{Model: llm}) over the raw h.reg.LLM(ref) in compactionFor, mirroring the ADK default but bypassing this wrapper, so summarization usage never lands in the turn clock. (The configured-summarizer path already uses the raw llm and is unaffected.)

🟠 IMPORTANT — Summarizer usage is accumulated into the running turn's clock when `compaction.summarizerModel` is unset. This wrapper wraps every GenerateContent flowing through the root `selectorModel`. With the default config (compaction on, 20k-token threshold, no `summarizerModel`), `compactionFor` (`internal/harness/harness.go:1200-1235`) leaves `out.Summarizer` nil, so ADK's runner installs its default LLMSummarizer **over the root agent's model** — this `selectorModel` — (`runner/runner.go:172-201`: `m := llminternal.Reveal(llmAgent).Model; NewLLMSummarizer(LLMSummarizerConfig{Model: m})`). The summarizer calls `s.model.GenerateContent(...)` directly (`llm_summarizer.go:232`), so `m.clock.addUsage` adds the summarizer's prompt/cached tokens and call duration to the current turn, and the new "prompt complete" log reports inflated `prompt_tokens`/`cached_tokens`/`model_time` and a wrong `cache_hit_ratio` whenever compaction fires mid-turn. Verified this session: `sed -n '172,201p' /home/paseo/go/pkg/mod/google.golang.org/adk/v2@v2.5.0/runner/runner.go` and `sed -n '232p' .../session/compaction/llm_summarizer.go` (direct GenerateContent, no callbacks), plus `grep -n 'out.Summarizer' internal/harness/harness.go` (`compactionFor` sets it only when `SummarizerModel != ""`). Fix: when `SummarizerModel == ""`, install `compaction.NewLLMSummarizer(compaction.LLMSummarizerConfig{Model: llm})` over the raw `h.reg.LLM(ref)` in `compactionFor`, mirroring the ADK default but bypassing this wrapper, so summarization usage never lands in the turn clock. (The configured-summarizer path already uses the raw llm and is unaffected.)
- Summarizer usage can no longer inflate the turn clock: compactionFor
  always builds the summarizer over the raw registry model (the session
  model when no summarizerModel is configured), never letting ADK default
  to its installation over the wrapped selectorModel.
- Preemption now counts queued prompts instead of a one-shot flag, so a
  prompt queued behind an already-preempting prompt still preempts the
  turn that runs ahead of it, and the running turn yields at the next
  safe boundary whenever any prompt is waiting.
Author
Collaborator

Resolutions (head now b08bd15)

🟠 IMPORTANT — summarizer usage in the turn clock (comment 873): fixed. compactionFor now always installs the summarizer over the raw registry model — the session's own model when no summarizerModel is configured, mirroring what ADK would have defaulted to but routed past the selectorModel wrapper. Summarization calls can no longer land in m.clock.addUsage, so prompt_tokens/cached_tokens/model_time/cache_hit_ratio in the prompt complete log are the turn's own. The compaction test that previously asserted Summarizer == nil for the unset case was updated to assert the new contract (non-nil, over the session model).

🟡 SUGGESTION — one-shot preemption flag (comment 874): fixed. preemptPending bool is replaced with queuedPrompts int: every prompt increments at entry (before blocking on the turn lock) and retires its own entry via acquiredTurnSlot once it holds the slot. The running turn yields at the next safe boundary whenever queuedPrompts > 0, still never while a tool call is in flight. A prompt queued behind an already-preempting prompt now preempts the turn that runs ahead of it instead of silently waiting for it.

Question — trailing system message against real providers: honest answer: not yet exercised against Melious or the local llama.cpp endpoint — every e2e test in this PR runs against a fake server that accepts anything, and I cannot reach either real provider from this container. The plan flags this as the remaining verification step. The concern is real (OpenAI's Responses API expects system instructions first; a strict server may reject or ignore a trailing system input_text). Two ways forward, your call: (1) run one real prompt with the current build and watch whether the provider errors or the model still sees the clock block; (2) if it misbehaves, switch the clock to a trailing text part on the last user/tool content (role-user, accepted anywhere) — same cache property, slightly worse semantics (looks like a user message). I'd suggest verifying first before changing placement.

Verification after both fixes: go build, full go test ./..., racedetector test ./internal/harness/, golangci-lint — all clean.

## Resolutions (head now `b08bd15`) **🟠 IMPORTANT — summarizer usage in the turn clock (comment 873):** fixed. `compactionFor` now always installs the summarizer over the **raw** registry model — the session's own model when no `summarizerModel` is configured, mirroring what ADK would have defaulted to but routed past the `selectorModel` wrapper. Summarization calls can no longer land in `m.clock.addUsage`, so `prompt_tokens`/`cached_tokens`/`model_time`/`cache_hit_ratio` in the `prompt complete` log are the turn's own. The compaction test that previously asserted `Summarizer == nil` for the unset case was updated to assert the new contract (non-nil, over the session model). **🟡 SUGGESTION — one-shot preemption flag (comment 874):** fixed. `preemptPending bool` is replaced with `queuedPrompts int`: every prompt increments at entry (before blocking on the turn lock) and retires its own entry via `acquiredTurnSlot` once it holds the slot. The running turn yields at the next safe boundary whenever `queuedPrompts > 0`, still never while a tool call is in flight. A prompt queued behind an already-preempting prompt now preempts the turn that runs ahead of it instead of silently waiting for it. **Question — trailing system message against real providers:** honest answer: not yet exercised against Melious or the local llama.cpp endpoint — every e2e test in this PR runs against a fake server that accepts anything, and I cannot reach either real provider from this container. The plan flags this as the remaining verification step. The concern is real (OpenAI's Responses API expects system instructions first; a strict server may reject or ignore a trailing system `input_text`). Two ways forward, your call: (1) run one real prompt with the current build and watch whether the provider errors or the model still sees the clock block; (2) if it misbehaves, switch the clock to a trailing text part on the last user/tool content (role-user, accepted anywhere) — same cache property, slightly worse semantics (looks like a user message). I'd suggest verifying first before changing placement. Verification after both fixes: `go build`, full `go test ./...`, `racedetector test ./internal/harness/`, `golangci-lint` — all clean.
sam merged commit 9b014ab1f1 into main 2026-10-04 13:52:06 +00:00
sam deleted branch feat/time-awareness 2026-10-04 13:52:06 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sam/brainless!23
No description provided.