Overview
The evaluation tiers that check boxer's claims against real agents and runtimes, and what each costs.
boxer's evaluations run real coding agents against real runtimes to check what unit tests cannot. They test that:
- a harness's commands run in the sandbox
- each integration level holds
- each worktree gets its own sandbox
- an agent finds its own dev server
The latest two matrix passes, on 2026-09-25, scored 99.4% of weighted checks by default and 98.7% in URLs mode. Results has every tier's latest numbers.
Tiers
| Tier | What it asks | Cost |
|---|---|---|
| smoke | Does every configuration path work on each backend? | free, minutes |
t1 | Does each integration work, with a scripted model? | free, fast |
t2 | The same, with real models | cents |
adherence | When boxer only asks, how often do agents comply? | cents |
flow | Can an agent develop against a dev server in the sandbox? | ~$0.20 |
sdlc | Twelve development lifecycles in parallel worktrees | ~$0.30 |
matrix | The same workload at every integration level | $2 to $3 |
An integration level is how boxer routes a harness's commands into the sandbox: a hook rewrite, a PATH shim, or running the harness inside the guest. Enforcement lists them.
Run the tiers
make smoke # one backend; BOXER_BACKEND=docker make smoke for another
make eval-t1 # no credentials needed
make eval-matrix # needs AI_GATEWAY_API_KEYEvaluation strategy
The layers, the oracle, the rules, and the bugs they caught.
Results
The latest numbers by level, harness and backend, with their conditions.
The matrix tier
The workload, the thirteen claims, and how a cell is scored.
Benchmarks
What boxer costs, when each number was measured, and the withdrawn claims.