v1.3.21

fetch_page Filtering Benchmark

Last updated


Measured results of fetch_page before (old) and after (new) the v1.3.17 change that filters each page against the current request with the decision model.
The comparison answers three questions:
  1. Relevance: are pages unrelated to the task excluded?
  2. Content filtering: are unrelated parts removed before returning, while the key content survives?
  3. Reduction: how much less content actually reaches the agent?

Summary


MetricOldNew
Correct page-relevance decisions19 / 2222 / 22
Key content kept for related tasks11 / 15 (plus 1 JSON cut into invalid JSON)15 / 15
Content sent for related tasks (unfiltered 3,130 KB)595 KB, 81.0% less320 KB, 89.8% less
Content sent for unrelated tasks104 KB2.2 KB (notice only)
Average time: regular pages5.14 s5.42 s
Average time: pages over 1 MiB15.2 s20.7 s

Most of the old version's reduction comes from a fixed 100 KiB cut and from hiding related pages it misjudged as unrelated, so it is paid for with lost information. The new version's reduction comes from dropping blocks unrelated to the task, and every key item is kept.

How the two versions work


OldNew
Stored in cacheResult after noise removal and truncationOriginal content with only noise (ads, navigation and the like) removed
Long pagesCut at 100 KiBFiltered block by block against the task; only related blocks are returned
Page relevanceWhole page judged from its opening only; markdown onlyShort pages judged as a whole; long pages judged block by block and excluded only when no block is related; same rules for markdown and json
Over 1 MiBCut at 100 KiBThe first 1 MiB is checked and only blocks that directly answer the task are kept; if none, the agent is told the page is too large and to download it
Error pagesDetected by titleDetected by the actual HTTP status and the title

In the new version, task-based filtering runs right before returning, so freshly fetched content and content read from the cache go through the same path. The same page fetched for different tasks is filtered separately for each.

Cache and re-filtering


The new version separates what is stored from what is returned:
  1. The cache stores the original content: after fetching, the decision model removes only noise such as ads and navigation (markdown, keep_links off, content up to 1 MiB), and the rest is stored in ToriiDB as is. The cache never holds task-filtered results or withheld notices.
  2. No cache entry: the page is fetched and stored as in point 1, then filtered from that original content against the current task by the decision model and returned.
  3. Cache hit: when the same URL (same parameters) is requested again, the original content is read from the cache and filtered again against the current task by the decision model before returning.

So when different tasks request the same URL, none of them reuses the previous task's filtering. A page withheld as unrelated for one task is still available to the next related task.
Measured on English Wikipedia: Taiwan, two tasks in a row in one session:
FormatTaskSourceResultTime
markdownhow to bake sourdough breadfetchwithheld, 166 B7.6 s
markdowncapital of Taiwancachefiltered to 45 KB, includes Taipei0.7 s
jsonhow to bake sourdough breadfetchwithheld, 160 B7.2 s
jsoncapital of Taiwancachefiltered to 113 KB, includes Taipei, valid JSON1.4 s

After both calls the cache still holds the full original page (markdown 229 KB, json 572 KB). Reading from the cache saves the fetch; the remaining time is the task-based re-filtering.

Relevance: unrelated pages are excluded


Each page is paired with one related task and one unrelated task. A related task should not be hidden; an unrelated task should get a withheld notice.
PageRelated: oldRelated: newUnrelated: oldUnrelated: new
MDN Array.mapreturnedreturnedexcludedexcluded
Python stdtypeswrongly hiddenreturnedexcludedexcluded
pkg.go.dev net/httpreturnedreturnedexcludedexcluded
English Wikipedia: Taiwanreturnedreturnedexcludedexcluded
Chinese Wikipedia: Japanreturnedreturnedexcludedexcluded
Effective Gowrongly hiddenreturnedexcludedexcluded
RFC 9112returnedreturnedexcludedexcluded
ECMAScript (1.8 MB)returnedreturnedexcludedexcluded
example.comreturnedreturnedexcludedexcluded
GitHub go-rodreturnedreturnedexcludedexcluded
WWII (json)returnedreturnednot excluded (102 KB)excluded

  • Old version hid related pages: it judged the whole page from its opening. The maxsplit part of Python stdtypes and the defer part of Effective Go are not near the top, so both pages were hidden and the agent had to look elsewhere.
  • Old version let an unrelated page through: it never judged json relevance, so an unrelated task still received 102 KB of WWII content.

Content filtering: unrelated parts are removed before returning


Related tasks only. Each task lists 1-2 items that must appear in the result to count as kept.
Page / taskKey contentOldNew
MDN Array.map / what map returnsnew array, callbackFn2/2 full page2/2 full page
Python stdtypes / maxsplitmaxsplit, sep0/2 hidden2/2 filtered
net/http / Client timeoutTimeout, Client2/2 full page2/2 filtered
English Wikipedia: Taiwan / capitalTaipei1/1 truncated1/1 filtered
Chinese Wikipedia: Japan / capital東京1/1 truncated1/1 filtered
Effective Go / deferdefer, LIFO0/2 hidden2/2 filtered
RFC 9112 / chunked formatchunk-size, last-chunk2/2 full page2/2 filtered
ECMAScript / Number typeNumber type1/1 opening only1/1 answer blocks only
example.com / purposedocumentation1/1 full page1/1 full page
GitHub go-rod / what rod isDevTools Protocol1/1 full page1/1 full page
WWII json / invasion of Poland1 September 19391/1 but invalid JSON1/1 valid JSON

  • The new version keeps all 15 key items. 8 long pages were filtered against the task; 3 short pages were short enough to return whole.
  • The old version did no task-based filtering, only a fixed 100 KiB cut. When the answer sits further down it is lost; the 2 wrongly hidden pages lost 4 key items.
  • The old ECMAScript result contains the words Number type, but only as a table-of-contents mention near the top; the new version's 1.3 KB is the definition itself.

Reduction: how much content is removed


The baseline is the unfiltered fetch_page output, which is what the cache stores (frontmatter plus the body after noise removal). ECMAScript is over 1 MiB, so its full page size is the baseline. Related tasks only; for unrelated tasks the new version always returns a notice of about 150-270 B.
PageUnfilteredOldOld reductionNewNew reduction
MDN Array.map5.4 KB5.4 KB0.5% more5.4 KB0%
Python stdtypes169 KB185 B99.9% (hidden)9.0 KB94.6%
net/http84 KB84 KB0%17 KB79.6%
English Wikipedia: Taiwan224 KB100 KB55.4% (truncated)42 KB81.1%
Chinese Wikipedia: Japan146 KB100 KB31.6% (truncated)19 KB87.1%
Effective Go96 KB197 B99.8% (hidden)13 KB86.9%
RFC 911288 KB88 KB0%35 KB60.2%
ECMAScript1,806 KB100 KB94.5% (truncated)1.2 KB99.9%
example.com1.5 KB1.5 KB1.7% more1.5 KB0%
GitHub go-rod2.2 KB2.1 KB2.9%2.1 KB4.1%
WWII json435 KB100 KB77.0% (invalid JSON)167 KB61.5%
Total3,130 KB595 KB81.0%320 KB89.8%

  • At roughly 4 bytes per token: unfiltered is about 780K tokens, new about 80K, old about 150K.
  • MDN and example.com are already short and need no filtering, so the new reduction is 0%. The 4.1% for go-rod comes from noise removal differing slightly between two fetches. Old short pages come out 1-2% larger because of the Web page content: wrapper, which was Removed in v1.3.17.
  • WWII json is still 167 KB, the largest new output. A filtered json tree carries no filter header, so the agent receives it without a note that blocks were dropped.

Timing


ScopeOld averageNew averageDifference
Regular pages (20 cases, excluding ECMAScript)5.14 s5.42 s+0.28 s
Over 1 MiB (2 ECMAScript cases)15.2 s20.7 s+5.5 s

Times include fetching in Chrome (every call used force and skipped the cache), so most of the time is the fetch itself. The extra time in the new version is block-by-block judging against the task; pages over 1 MiB cost the most because the first 1 MiB is judged in small batches.

Method


  • Old: commit b45377ab, the fetch_page behaviour released in v1.3.16; dependencies are the go.mod releases (go-browser v0.3.4).
  • New: the working tree that became v1.3.17, using local go-browser (with the keep_links fix, since released as v0.3.5), go-llm-router and go-pkg through go.work.
  • Both versions were called through the tool entry toolregister.Dispatch("fetch_page") with force: true; decision model typesafe@jev-latest; run one after the other so they did not compete for Chrome or the decision model.
  • Each case ran once. The decision model's answers can vary between runs, so borderline blocks may come and go.
  • Key content was checked by string match, which shows the content is present but not that answers are good. The sample is 11 pages, weighted toward English technical documentation.
  • The reduction baseline is the output after readability extraction and noise removal, not the page's raw HTML. It was fetched separately, so noise removal may differ slightly from the test run.
  • Source write-up: doc/decision/fetchPage.md.
中文