TokenSave's 288-run study: asking for subagent search left recall almost unchanged and raised the Opus bill by up to 221%
Claude Code 2.1.318 added subagent effort (`CLAUDE_CODE_SUBAGENT_EFFORT`). TokenSave (dataset under johnisking; Japanese write-up by @Yunany90) ran 288 pre-registered headless trials on 12 “find every call site” questions across 6 OSS repos, Haiku 5.5 / Opus 5.5, no-instruction vs forced Explore subagents at low/medium/high. Result: recall stayed ~97–100% (<2 pt gaps); cost rose Haiku +74–116%, Opus +143–221%; with no instruction, 0/72 runs called a subagent. We recomputed the CSV: 288 rows, balanced cells, Opus C2/C0 mean cost ≈3.21× (+221%), C0 subagent_calls sum 0. We did not re-bill a live Claude Code session.
Visit site Claude Code subagent effort (TokenSave study) ↗- Pricing
- Paid
- Region
- Global
- Where it's from
- Overseas
- Added
- Nov 7, 2026
- Tested on
- Nov 7, 2026
- Version we used
- token-cost-by-language @ 1acf67e; Claude Code 2.1.318 per study; dataset DOI 10.5281/zenodo.24014489
- Categories
- CodingProductivityResearch
- Original author
- TokenSave / 조재현 @Yunany90 ↗
- Our test
- Partly tested
- Cost / time
- Dataset and write-ups are free (CC BY 4.0). Your own Claude Code usage still bills via Anthropic; this study shows casual subagents can sharply raise per-run cost.
The task
When doing ordinary “find every call site” searches in Claude Code, decide whether to force Explore subagents in the prompt and whether to hand-tune subagent effort.
Steps
- Read the results page or repo: https://tokensave.app/data/claude-subagent-effort and GitHub `claude-subagent-effort/` (PREREG.md, questions.json, CSV/JSON, scorers). License CC BY 4.0.
- For everyday code search: state what to find; don't casually add “use subagents / Explore”. With no instruction the harness almost never calls them (0/72 here).
- Reserve subagents for large monorepos or tasks you personally A/B for cost vs recall; don't default `CLAUDE_CODE_SUBAGENT_EFFORT` to medium/high.
- If you drive headless/`claude -p` with subagents: 2.1.318 emits multiple result events—join all of them; scoring only the first tanks accuracy to single digits.
- To reproduce: pin repo commits, pre-register answer keys, allow read/search tools only; watch the 5-hour subscription limit creating empty runs.
Prompt
In this repo, find every place RequestAbortedError is created; list file paths and enclosing function/method names. Use Grep and Read yourself and write the final answer—do not launch subagents.
What the author says
TokenSave / @Yunany90: 288 pre-registered runs show forced subagent search barely moves recall (~97–100%) while raising Haiku cost +74–116% and Opus +143–221%, with ~2–3× latency; subagents bill at the main model; with no instruction none were started. Chinese amplifiers include @LiLensss.
Our test
Cloned johnisking/token-cost-by-language (commit 1acf67e), read claude-subagent-effort/README.md and PREREG.md, and recomputed claude-subagent-effort.csv/.json in Python. Cross-checked Qiita @Yunany90 and the tokensave.app page. No Claude Code install, no re-run of the 288 headless jobs, no subscription-limit behaviour check.
What we found when we tried it:CSV 288 rows; 144 each Haiku/Opus; 36 per condition. Question-mean recall: Haiku C0/C1/C2/C3 ≈ 0.982/0.970/0.985/0.987; Opus ≈ 0.997/0.993/0.990/1.000. Mean cost: Haiku $0.0056/$0.0098/$0.0105/$0.0121; Opus $0.1589/$0.3861/$0.5093/$0.4825. C0 subagent_calls sum 0; Opus C2 vs C0 ≈ +221%. Matches the published figures.
How to get in
Public GitHub / Zenodo / Hugging Face / tokensave.app; Japanese Qiita longform; Chinese summary via @LiLensss. Needs GitHub access.
Who it suits
Daily Claude Code users who care about bill and latency; people wrapping Claude Code in CI/headless scripts.
Pitfalls
- Questions are mid-size “find the full list” tasks where base models were near ceiling—huge monorepos may differ.
- One prompt wording and Claude Code 2.1.318 only; re-test after version changes.
- Headless multi-result events are easy to score wrong—read the scoring notes before reproducing.
- We recomputed the public CSV/JSON only; we did not re-spend API on 288 runs.
More in this category
- A demo video per docs page that flags itself when the UI changesVideoScreenplayClaude CodeTool free; uses your agent's quota◐ Partly testedOverseas
- AI backend code that guards against races, leaks and bad migrationsCodingbackendskillsClaude CodePostgreSQLFree; about 2× tokens per task◐ Partly testedOverseas
- An afternoon of AI work, a 3D action game you play in the browserCodingClaude CodeTripoJimengSkill free; Claude quota plus asset tools◐ Partly testedChina
- Keep a long-running AI-built project from turning into a messCodingClaudeCodexoil-delegateNeeds Claude and Codex plans◐ Partly testedChina
- Use a web app once and get an AI write-up of how it worksResearchREAClaude CodeChromeFree tools; needs Claude or Codex◐ Partly testedOverseas