feat(tools): add an always-visible built-in fetch tool #20
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
sam/brainless!20
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/builtin-fetch-tool"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
An agent could only reach the web through an MCP server, so a session with no web-fetching server could not read a URL at all. This adds a
fetchtool to theinternal/toolspackage next to the built-in file tools, registered the same way: always visible, so its schema rides in every request with notoolsdiscover/load round trip.Modelled after the reference MCP fetch server (https://github.com/modelcontextprotocol/servers/tree/main/src/fetch).
What the tool does
fetch(url, maxLength?, startIndex?, raw?, timeout?):http/httpsURL. Anything else is rejected before a request goes out.raw: truereturns the body as fetched, and a non-HTML content type is always returned raw behind a prefix saying so.maxLengthdefaults to 5000 characters, a shortened result ends by naming thestartIndexto continue from, and a start at or past the end is reported as such. Indices count characters, not bytes, so a window never splits a multi-byte character. A body is read up to 8 MiB.timeoutseconds (default 30, at most 300), applied as one deadline over the robots.txt check and the page together. The HTTP client itself has no timeout, so an earlier caller deadline and a canceled turn still win.robots.txtis fetched and evaluated first, per RFC 9309 (longest matching pattern wins,Allowwins a tie), so a path the site disallows is refused. Following the reference server: a robots.txt answering 401/403 refuses the fetch, any other 4xx means there are no rules, and a robots.txt that cannot be read at all fails the fetch rather than fetching unchecked.brainless/0.0.1 (+https://git.norobo.chat/sam/brainless), follows redirects, and honors the proxy environment.Notes for review
fileResult→toolResult, since the fetch tool now shares it (a pure rename of an unexported type; verified identical after the rename).go mod tidyalso bumped the indirectmattn/go-isatty0.0.20 → 0.0.22.go-shiori/go-readabilitybefore the conversion is the follow-up if the first window should be article text.Verification
go test ./...passes; new tests cover conversion (incl. tables and absolute links), windowing and rune-safe slicing, content-type sniffing, raw mode, status errors, the timeout argument and a hanging server, robots allow/deny/failure modes, and RFC 9309 pattern matching.racedetector test ./internal/tools/ ./internal/harness/— no data races.golangci-lint run ./...— 0 issues.go.devanden.wikipedia.org: robots checked, markdown returned, absolute links, truncation hint. That smoke test was a throwaway and is not part of the commit.README
The new "Built-in fetch tool" section documents the arguments, the robots.txt rules, the time bound, and the readability gap.
Review of feat(tools): add an always-visible built-in fetch tool
Reviewed with the samber/cc-skills-golang pass order (style/naming/docs → correctness/safety/errors/concurrency → security/deps → tests/perf/observability/modernize). The diff was read from the PR head
a0c0465via the Forgejo API; every claim below was verified in this session against the repo's own toolchain (go1.27.1) in a scratch clone at the head commit.Baseline (all green)
go build ./...,go vet ./...,go test ./...— pass (run at head).gofmt -lclean;golangci-lint run ./...— 0 issues.govulncheck ./...— 0 vulnerabilities in code paths; the 3 module-level findings are all in pre-existinggolang.org/x/crypto@v0.55.0(untouched by this PR). Noreplacedirectives;html-to-markdown/v2 v2.5.2anddom v0.3.1are current, require Go 1.25 (module is 1.27.1), not deprecated.acp.ToolKindFetchexists in coder/acp-go-sdk (go doc),slices/strings.SplitSeq/min/maxexist in this toolchain,omitzeroandt.Context()are fine on go 1.27.Findings
One blocking, one important, four suggestions — all with reproduced evidence. The two redirect findings were reproduced with temporary tests run against the real code at head (test files removed afterwards; command outputs quoted inline). RFC 9309 pattern semantics (
/privateprefix-matching/privateer, longest-match/Allow-tie rules, §2.3.1.2 redirect handling of robots.txt) were checked against the RFC text and are implemented correctly; the user-agent matching nit (comment #4) and the 5xx robots.txt handling (comment #3) deviate from §2.2.1 and §2.3.1.4 respectively.The rename
fileResult→toolResultis a clean, complete rename (no leftover references). Tests are thorough, table-driven where it matters, uset.Helper()/t.Parallel()/t.Context(). Concurrency is clean:Fetchis stateless,*http.Clientis documented safe for concurrent use, context is propagated end to end, and the per-call converter inhtmlToMarkdownis a documented, deliberate choice. Error handling wraps with%wand never swallows. Naming, doc comments, and README are excellent.@ -165,0 +198,4 @@of holding the turn open, and a canceled turn still stops it immediately.Like the file tools it takes the caller's context, so a cancelled turn stops thefetch, and it runs without a permission prompt: the ACP client is trusted, the🟡 SUGGESTION — this paragraph documents the trusted-ACP-client posture, but the fetch tool is the first outbound network channel exposed to the model, and it can reach internal addresses (localhost, RFC1918, cloud metadata 169.254.169.254) while page content — an untrusted prompt-injection vector — is fed straight back to the model. If brainless can ever run in an environment where metadata or internal services matter, consider documenting this exposure in the security section or blocking loopback/private/metadata ranges. This is a design consideration, not a claim that the current trust model is wrong.
@ -0,0 +140,4 @@return toolResult{}, err}maxLength, err := fetchMaxLength(args.MaxLength)🟡 SUGGESTION — maxLength and startIndex are validated only after
checkRobotshas already made a network request (lines 139-150). An invalid window argument therefore costs a robots.txt round-trip before the tool fails. MovefetchMaxLengthand the startIndex check abovecheckRobotsso all argument validation happens before any I/O.@ -0,0 +163,4 @@switch {case isPageHTML && !args.Raw:content, err = htmlToMarkdown(page.body, target)🔴 BLOCKING — relative links are resolved against the original URL, not the URL the page actually came from after redirects.
getfollows redirects (defaulthttp.Client: up to 10, cross-host), so the body inpageis the final response, buttargetis still the original URL when it is passed tohtmlToMarkdownhere and to the "Contents of" prefix at line 178. Any redirected page — http→https, www→apex, moved paths — gets wrong absolute links, breaking the tool's documented promise ("relative links are made absolute against the fetched page").Reproduced with a temporary test at head (removed after):
/old/page→ 302 →http://127.0.0.1:44225/new/pageserving<a href="/real-target">produced output[link](http://127.0.0.1:46805/real-target)— resolved against the original host, not the final host. (Also verifiedgo doc net/http.Response: "Request is the request that was sent to obtain this Response", i.e. the final request after redirects.)Fix: have
getreturnresp.Request.URLand pass that tohtmlToMarkdownand the output prefix instead of the originaltarget.@ -0,0 +297,4 @@// at all is an error, because the tool does not fetch what it cannot check; 401// and 403 mean autonomous fetching is not allowed; any other 4xx means there is// no robots.txt to obey; and otherwise the rules decide.func (f *Fetch) checkRobots(ctx context.Context, target *url.URL) error {🟠 IMPORTANT — robots.txt is evaluated only for the original URL, so a redirect bypasses it.
checkRobotsbuilds the robots URL fromtargetand evaluates rules againsttarget.RequestURI(), but the page fetch (f.get, line 274) then follows up to 10 redirects (defaulthttp.Clientpolicy, cross-host included). A site that 302s/public→/private/secretis fetched even when its robots.txt saysDisallow: /private.Reproduced with a temporary test at head (removed after): robots.txt
User-agent: * Disallow: /private,/public→ 302 →/private/secret;fetch(/public)returned"Contents of http://127.0.0.1:46583/public: SECRET DISALLOWED CONTENT"with no error. The reference server (httpxfollow_redirects=True, single robots check) shares this gap, but the README and the tool description promise "a path the site disallows is refused".Fix: set
http.Client.CheckRedirectto re-runcheckRobotsagainst each redirect target (or follow redirects manually), or document the limitation.@ -0,0 +308,4 @@switch {case page.status == http.StatusUnauthorized || page.status == http.StatusForbidden:return fmt.Errorf("fetch %s: %s returned status %d, so autonomous fetching is not allowed", target, robots, page.status)case page.status >= http.StatusBadRequest:🟡 SUGGESTION — a robots.txt answering 500-599 is treated as "no rules" (
page.status >= http.StatusBadRequestcovers 5xx), but RFC 9309 §2.3.1.4 says a server error means robots.txt is undefined and the crawler "MUST assume complete disallow". This also contradicts the doc comment above ("a robots.txt that cannot be read at all fails the fetch rather than fetching unchecked"). Condition: the origin's robots.txt returns a transient 5xx — the tool then fetches pages whose policy is unknown. The reference server fails open here too (it parses the 5xx body as robots.txt), so this is a fidelity note, not a parity break.Fix: return an error for
status >= 500(fail closed per RFC), or at minimum fix the comment to say "any other 4xx or 5xx".@ -0,0 +144,4 @@}product, _, _ = strings.Cut(strings.ToLower(product), "/")if product == agent || strings.HasPrefix(product, agent) {🟡 SUGGESTION —
strings.HasPrefix(product, agent)matches prefix tokens: aUser-agent: braingroup would be applied to thebrainlessproduct token. RFC 9309 §2.2.1 requires case-insensitive matching of the product token against the group's identifier — an exact identifier match, not a prefix. Practical impact is low whilefetchProductTokenis the constant"brainless", but the file claims RFC 9309 compliance andTestRobotsAllowdoes not cover this.Fix: drop the
|| strings.HasPrefix(product, agent)arm and matchproduct == agentonly.a0c0465b69edb6a12cb3All six review findings are resolved on head
edb6a12(amended commit, force-pushed).#729 — BLOCKING: relative links after redirects →
getnow returns the effective URL (resp.Request.URL, the final request after redirects) andfetchuses it for both the markdown domain (WithDomain) and theContents of …:prefix. AddedTestFetchResolvesLinksAgainstFinalURL(two servers:/old/page302 →dest/new/pagewith<a href="/real-target">; output resolves againstdest, prefix namesdest).#730 — IMPORTANT: redirects bypass robots.txt → the page client now has
CheckRedirectthat re-runscheckRobotsagainst every redirect target, aborting the fetch when a hop is disallowed. A separate redirect-free client fetches robots.txt itself, so a redirecting robots.txt cannot loop back into another check. AddedTestFetchRedirectCannotBypassRobots(/public302 →/private/secretwithDisallow: /private; fetch now fails with the robots refusal). The tool description now says "at every redirect the page takes".#731 — SUGGESTION: robots.txt 5xx → a 5xx robots.txt now fails the fetch closed (RFC 9309 §2.3.1.4, "assume complete disallow"), alongside 401/403, with the reason in the error; the doc comment and README updated. Covered by a new case in
TestFetchRobotsTxtFailures.#733 — SUGGESTION: arg validation after I/O →
fetchMaxLength, thestartIndexcheck, and the timeout check now all run before any request, so a bad window or timeout costs no robots.txt round-trip.#732 — SUGGESTION: prefix user-agent matching →
robotAgentMatchnow matches the product token exactly (RFC 9309 §2.2.1), dropping theHasPrefixarm; aUser-agent: braingroup no longer applies tobrainless. Covered inTestRobotsAllow.#734 — SUGGESTION: outbound exposure / prompt injection → documented in the README's fetch section ("Exposure."), stating the tool can reach internal addresses (localhost, RFC 1918, cloud metadata) and that page content is an untrusted prompt-injection vector handed to the model, safe only under the trusted-ACP-client posture; blocking ranges is left as a future restriction if the deployment requires it.
Verification:
go test ./...,racedetector test ./internal/tools/ ./internal/harness/(no races),golangci-lint run ./...— all green.