v1.3.21

read_files Filtering Benchmark

Last updated


Measured results of read_files without a decision model and with the v1.3.21 filtering that checks each read against the current request with the decision model.
The comparison answers three questions:
  1. Relevance: are files unrelated to the task withheld?
  2. Content filtering: are unrelated parts removed before returning, while the key content survives?
  3. Reduction: how much less content actually reaches the agent?

Summary


MetricWithoutWith
Unrelated files withheld0 / 1212 / 12
Related files wrongly withheld0 / 120 / 12
Key content kept for related tasks12 / 1212 / 12
Content sent for related tasks (342 KB)342 KB189 KB, 44.8% less
Content sent for related tasks, files of 16K characters or more (288 KB)288 KB134 KB, 53.3% less
Content sent for unrelated tasks (342 KB)342 KB2.2 KB (notice only)
Average time per read (24 reads)19 ms245 ms

Without a decision model every read returns the whole file, whatever the task. With one, unrelated files come back as a short notice, long related files lose the blocks that do not bear on the task, and short related files come back whole.

How the two modes work


WithoutWith
Files under 16K charactersReturned wholeJudged as a whole; unrelated files are replaced by a withheld notice, related files are returned whole
Files of 16K characters or moreReturned wholeSplit into blocks at blank lines and judged block by block; unrelated blocks are replaced by a row-range marker, and a file with no related block is replaced by a withheld notice
Getting the whole fileAlwaysRead again with full: true, which skips the filter
Writing after a filtered read—write_file rejects content that still contains a marker and asks for a full: true read

The marker names the rows it replaced, for example [unrelated: rows 120-188 not loaded; read it again with full: true to load the whole file], so the agent can see the gap and knows how to load it. Block judgments that fail or time out keep their blocks, and with no decision model configured every read behaves as in the "Without" column.

1. Relevance: unrelated files are withheld


Each file is paired with one related task and one unrelated task (How do I bake sourdough bread at home?). A related task should not be withheld; an unrelated task should get a withheld notice.
FileSizePathRelatedUnrelated
README.md26 KBblocksreturnedwithheld
doc/doc.md135 KBblocksreturnedwithheld
doc/architecture.md39 KBblocksreturnedwithheld
internal/agents/exec/toolCall.go34 KBblocksreturnedwithheld
internal/agents/exec/execute.go34 KBblocksreturnedwithheld
internal/runtime/tui/viewRender.go20 KBblocksreturnedwithheld
CLAUDE.md24 KBwhole filereturnedwithheld
configs/configs.go10 KBwhole filereturnedwithheld
internal/tools/file/boundary/boundary.go8.2 KBwhole filereturnedwithheld
internal/tools/file/relevance.go5.0 KBwhole filereturnedwithheld
go.mod4.7 KBwhole filereturnedwithheld
makefile1.8 KBwhole filereturnedwithheld

Sizes are bytes of the read_files output, row numbers included. CLAUDE.md is 24 KB but under 16K characters because most of it is Chinese, so it takes the whole-file path.

2. Content filtering: unrelated parts are removed before returning


Related tasks only. Each task lists one item that must appear in the result to count as kept.
File / taskKey contentWithMarkers
README.md / how to install Agenvoyinstall.shkept14
doc/doc.md / decision model call timeouttimes out after 3 secondskept17
doc/architecture.md / which official_guides layers are injected_vendor_kept6
toolCall.go / how the read_files cache is invalidated after a writeinvalidateReadFileCachekept12
execute.go / when alreadyCall is resetalreadyCall = makekept10
viewRender.go / how the TUI renders a write_file diffwriteDiffLinekept9
CLAUDE.md / why Go builds use -tags fts5FTS5kept, whole file—
configs.go / value of MAX_DECISION_STATE_RUNES20_000kept, whole file—
boundary.go / how ResolveIn resolves a pathfunc ResolveInkept, whole file—
relevance.go / the relevance size thresholdMIN_RELEVANCE_RUNESkept, whole file—
go.mod / which go-llm-router version is usedgo-llm-router vkept, whole file—
makefile / how the binary is builtfts5kept, whole file—

  • All 12 key items are kept. The 6 long files were filtered block by block and left 6-17 markers each; the 6 short files were returned whole.
  • Without a decision model all 12 items are present too, since nothing is ever removed; the difference is the size of what comes with them.

3. How much content is removed


The baseline is the full: true output, which is what every read returns without a decision model. Related tasks only; for unrelated tasks the decision model always returns a notice of about 170-210 B.
FileWithoutWithReduction
README.md26 KB9.6 KB63.3%
doc/doc.md135 KB59 KB56.7%
doc/architecture.md39 KB28 KB27.1%
internal/agents/exec/toolCall.go34 KB15 KB56.9%
internal/agents/exec/execute.go34 KB13 KB61.8%
internal/runtime/tui/viewRender.go20 KB10 KB47.8%
6 files under 16K characters55 KB55 KB0%
Total342 KB189 KB44.8%

  • At roughly 4 bytes per token: about 86K tokens without, about 47K with.
  • doc/architecture.md loses the least because its paragraphs are long and the task touches several of them.
  • doc/doc.md still returns 59 KB, the largest filtered output; the task matches many blocks across a 135 KB file.

Timing


ScopeWithout averageWith averageDifference
Files of 16K characters or more (12 reads)28 ms289 ms+261 ms
Files under 16K characters (12 reads)10 ms201 ms+191 ms

Without a decision model a read is a local file read. The extra time is the decision call: one call for a short file, and block batches run in parallel for a long one.

Method and limitations


  • Version: git bb0de320.
  • Both modes were called through the tool entry toolregister.Dispatch("read_files") in one test process: "Without" with full: true, which skips the filter exactly as a missing decision model does, and "With" with full: false; decision model typesafe@jev-latest; run one after the other.
  • Each case ran once. The decision model's answers can vary between runs, so borderline blocks may come and go.
  • Key content was checked by string match, which shows the content is present but not that the surrounding context is enough. The sample is 12 files from this repository, weighted toward Go source and English documentation.
  • The baseline is the read_files output with row-number prefixes, read with the default limit of 2,048 lines.
中文