Turn 1,270 of your own coding-agent sessions into a private eval set and find a setup at half the cost
Ibragim, who maintains SWE-rebench, had an agent read three months of his Claude Code / Codex / Pi sessions, picked episodes where the agent failed and he corrected it, turned them into 10 Harbor tasks with automated checks, and ran different models and harnesses. On his tasks, Pi + GPT-6.1 Sol solved everything at about half Codex's cost per task with the same model.
Visit site evals-from-sessions (Claude Code / Codex / Pi skill) + Harbor ↗- Pricing
- Free
- Region
- Global
- Where it's from
- Overseas
- Added
- Oct 18, 2026
- Tested on
- Oct 18, 2026
- Version we used
- evals-from-sessions SKILL.md 1.0.4 (pushed 2026-10-14); Harbor 0.24 syntax
- Original author
- Ibragim @ibragim_bad ↗
- Our test
- Partly tested
- Cost / time
- The skill (MIT) and Harbor are free; you pay for model calls during runs and the time to build tasks.
The task
Stop guessing when picking a coding agent: measure whether the pricier model, more reasoning or a plugin pays off on your own day-to-day tasks.
Steps
- Back up sessions: Claude Code in ~/.claude/projects/, Codex CLI in ~/.codex/sessions/, Pi in ~/.pi/agent/sessions/, Gemini CLI in ~/.gemini/tmp/. Work on the copy and scan it with gitleaks first.
- Install the skill: npx skills add ibragim-bad/evals-from-sessions (or as a Claude Code / Codex plugin). Needs Docker, git, Python 3.11+, Harbor (uv tool install harbor) and gitleaks 8.
- Have the agent read the sessions and list candidates in candidates.md: goal, files touched, where the agent went wrong, how it ended, and whether it can be auto-verified. First human checkpoint.
- Write the spec (provenance.md): requirements, failure modes the tests must catch, and what the agent must discover itself. Approve it (second checkpoint) before the Harbor task is built.
- Validate the verifier: do-nothing fails, the reference passes 5/5, a near-miss fails, an alternative correct solution passes; then run the leak probe and harbor check.
- Run setups with harbor run -p tasks/ -a <agent> -m <model> -k 3 and compare all-3-runs pass rate, cost and time; have another agent audit trajectories for loopholes and false verdicts, fix, re-run.
What the author says
The author (2026-10-16) says he started with 15 tasks from ~1,270 sessions and kept 10 after 3–5 iterations, running each setup 3 times per task. Several setups solved everything, so he compared cost and speed: Pi + GPT-6.1 Sol at about $0.070 per task vs. $0.138 for Codex on the same model. He stresses this picks a setup for his own work, not an overall model ranking.
Our test
We didn't build any evals or reproduce the $0.070 vs. $0.138 comparison (no Docker or Harbor on our test machine). We only checked the skill repo and the guide links.
What we found when we tried it:Checked 2026-10-18: the GitHub repo ibragim-bad/evals-from-sessions is live, MIT-licensed, SKILL.md version 1.0.4, with Claude Code / Codex plugin manifests, a Harbor task template (instruction.md, task.toml, tests/test.sh, solution/solve.sh, Dockerfile) and leak-probe.sh. The template's test.sh always writes a reward file; leak-probe.sh checks commit, ref, remote, tag and hook counts in a network-less container and greps the whole image for strings unique to the reference solution. The guide page on ibragim.dev loads.
How to get in
English guide on X and on ibragim.dev; the repo is on GitHub.
Who it suits
Developers and teams spending real money on coding agents who want evidence before switching models or harnesses.
Pitfalls
- Session logs often contain API keys; scan with gitleaks before sharing an export.
- Some agents delete old sessions (Claude Code after 30 days by default), so back up early.
- Agents cheat: on tasks from public repos they may find the fix online or in git history. Strip future commits and the reference diff, and ask them in the prompt to solve honestly.
- Verifier bugs cut both ways: valid solutions marked wrong (e.g. exact error-string matching) and broken ones marked right (tests miss a requirement). Both skew the comparison.
- His dead end: a git cleanup that took him 90 minutes was solved in the sandbox by one git merge --ff-only. Drop tasks that turn out trivial.
More in this category
- A demo video per docs page that flags itself when the UI changesVideoScreenplayClaude CodeTool free; uses your agent's quota◐ Partly testedOverseas
- AI backend code that guards against races, leaks and bad migrationsCodingbackendskillsClaude CodePostgreSQLFree; about 2× tokens per task◐ Partly testedOverseas
- An afternoon of AI work, a 3D action game you play in the browserCodingClaude CodeTripoJimengSkill free; Claude quota plus asset tools◐ Partly testedChina
- Keep a long-running AI-built project from turning into a messCodingClaudeCodexoil-delegateNeeds Claude and Codex plans◐ Partly testedChina
- Use a web app once and get an AI write-up of how it worksResearchREAClaude CodeChromeFree tools; needs Claude or Codex◐ Partly testedOverseas