evals-from-sessions (Claude Code / Codex / Pi skill) + HarborPartly tested🌏 World

Turn 1,270 of your own coding-agent sessions into a private eval set and find a setup at half the cost

Ibragim, who maintains SWE-rebench, had an agent read three months of his Claude Code / Codex / Pi sessions, picked episodes where the agent failed and he corrected it, turned them into 10 Harbor tasks with automated checks, and ran different models and harnesses. On his tasks, Pi + GPT-6.1 Sol solved everything at about half Codex's cost per task with the same model.

Visit site evals-from-sessions (Claude Code / Codex / Pi skill) + Harbor ↗
Pricing
Free
Region
Global
Where it's from
Overseas
Added
Oct 18, 2026
Tested on
Oct 18, 2026
Version we used
evals-from-sessions SKILL.md 1.0.4 (pushed 2026-10-14); Harbor 0.24 syntax
Categories
CodingResearch
Original author
Ibragim @ibragim_bad ↗
Our test
Partly tested
Cost / time
The skill (MIT) and Harbor are free; you pay for model calls during runs and the time to build tasks.
Source
X @ibragim_bad ↗ibragim.dev ↗GitHub ibragim-bad/evals-from-sessions ↗

The task

Stop guessing when picking a coding agent: measure whether the pricier model, more reasoning or a plugin pays off on your own day-to-day tasks.

Steps

  1. Back up sessions: Claude Code in ~/.claude/projects/, Codex CLI in ~/.codex/sessions/, Pi in ~/.pi/agent/sessions/, Gemini CLI in ~/.gemini/tmp/. Work on the copy and scan it with gitleaks first.
  2. Install the skill: npx skills add ibragim-bad/evals-from-sessions (or as a Claude Code / Codex plugin). Needs Docker, git, Python 3.11+, Harbor (uv tool install harbor) and gitleaks 8.
  3. Have the agent read the sessions and list candidates in candidates.md: goal, files touched, where the agent went wrong, how it ended, and whether it can be auto-verified. First human checkpoint.
  4. Write the spec (provenance.md): requirements, failure modes the tests must catch, and what the agent must discover itself. Approve it (second checkpoint) before the Harbor task is built.
  5. Validate the verifier: do-nothing fails, the reference passes 5/5, a near-miss fails, an alternative correct solution passes; then run the leak probe and harbor check.
  6. Run setups with harbor run -p tasks/ -a <agent> -m <model> -k 3 and compare all-3-runs pass rate, cost and time; have another agent audit trajectories for loopholes and false verdicts, fix, re-run.

What the author says

The author (2026-10-16) says he started with 15 tasks from ~1,270 sessions and kept 10 after 3–5 iterations, running each setup 3 times per task. Several setups solved everything, so he compared cost and speed: Pi + GPT-6.1 Sol at about $0.070 per task vs. $0.138 for Codex on the same model. He stresses this picks a setup for his own work, not an overall model ranking.

Our test

We didn't build any evals or reproduce the $0.070 vs. $0.138 comparison (no Docker or Harbor on our test machine). We only checked the skill repo and the guide links.

What we found when we tried it:Checked 2026-10-18: the GitHub repo ibragim-bad/evals-from-sessions is live, MIT-licensed, SKILL.md version 1.0.4, with Claude Code / Codex plugin manifests, a Harbor task template (instruction.md, task.toml, tests/test.sh, solution/solve.sh, Dockerfile) and leak-probe.sh. The template's test.sh always writes a reward file; leak-probe.sh checks commit, ref, remote, tag and hook counts in a network-less container and greps the whole image for strings unique to the reference solution. The guide page on ibragim.dev loads.

How to get in

English guide on X and on ibragim.dev; the repo is on GitHub.

Who it suits

Developers and teams spending real money on coding agents who want evidence before switching models or harnesses.

Pitfalls

  • Session logs often contain API keys; scan with gitleaks before sharing an export.
  • Some agents delete old sessions (Claude Code after 30 days by default), so back up early.
  • Agents cheat: on tasks from public repos they may find the fix online or in git history. Strip future commits and the reference diff, and ask them in the prompt to solve honestly.
  • Verifier bugs cut both ways: valid solutions marked wrong (e.g. exact error-string matching) and broken ones marked right (tests miss a requirement). Both skew the comparison.
  • His dead end: a git cleanup that took him 90 minutes was solved in the sandbox by one git merge --ff-only. Drop tasks that turn out trivial.

More in this category