Strategy

The test layers from unit tests to live agents, what each proves, how results are judged, and the bugs the evals caught.

boxer is tested in layers, from unit tests against a fake backend up to live agents on real runtimes.

Each layer above the unit tests adds one real thing the layer below cannot check: a runtime, a harness, a live model or a Next.js workload.

The layers

layer	what it proves	what it cannot	cost
unit tests, fake backend	every code path, including every failure	that a runtime behaves like the fake	free, seconds
real-daemon tests	docker, podman and Apple `container` argv, JSON and exit codes	a runtime that is not installed, which skips by name	seconds
smoke	every configuration path, on each installed backend	that a harness cooperates	minutes per backend
t1	each harness's integration works end to end, with a scripted model	that a real model complies	minutes, free
t2	the same, with live models	how often a model complies unprompted	cents
adherence	whether an agent told only the brief does the right thing	that development works	cents
flow, sdlc, matrix	a real Next.js workload, at every integration level, in parallel worktrees	nothing below them	$0.19 to $3 a run
benchmarks	what it costs	whether it works, since they gate nothing	minutes

Smoke checks boxer's model of each runtime against the runtime. It runs the same cells on smolvm, Apple container, docker and podman. A cell a backend cannot support is skipped by name, using the capability table boxer itself reports, so a backend that claims a capability and then skips its cell is a finding.

t1 swaps the model for a scripted endpoint that speaks the Anthropic, OpenAI Chat and OpenAI Responses APIs, so a harness's hooks, its plugin and boxer run with no key and give the same result every time. It has 69 cells, one for every harness at every level it supports plus the orchestrators.

Adherence covers the cases where boxer only asks (a brief, a skill, an instruction file) and a live agent decides. Each scenario is a prompt that never mentions boxer:

scenario	the agent is asked to	it passes when
`brief`	run one command	it ran in the guest, with no correction
`recovery`	the same, after being refused once	it ran in the guest after one denial
`multistep`	install dependencies and run the tests	nothing intercepted ran on the host
`task`	run the test suite	it chose the repository's declared task
`prep`	read a file host-side `[prep]` wrote	prep ran once, with the guest's platform
`server`	fetch the running dev server's home page	it returned the page, reached by its URL

The matrix runs the same two Next.js tasks (add a page, install a dependency) through every integration level, harness and orchestrator, three at a time, each in its own worktree, sandbox and automatic port. Each cell scores thirteen weighted claims, described in the matrix tier. With BOXER_EVAL_URLS=1 every sandbox also gets a portless name and the prompt stops giving the agent its address.

What the oracle looks at

The oracle is the code that scores a cell. It ignores the model's own report of what it did. It reads evidence the agent cannot fake:

A canary. The command touches a file with a run-specific name. If it exists on the host, the command ran on the host, whatever the agent says.
The guest. A platform probe written by the work itself, corroborated by a dependency that must be in package.json and by the level's own signature in boxer's trace.
The trace. Every hook invocation and its answer, so a command that was allowed silently is distinguishable from one that was rewritten.
The host afterwards. Sandboxes, routes and scratch left behind are findings.
The Host header. The server scenario's page answers with the host it was reached by, so a URL and a looked-up port are told apart.

The rules

Every skip prints its reason. A missing credential, a runtime that is not installed, or a harness that cannot do a level each skips with the reason.
Two models before a harness is blamed. One model failing a cell among passes is that model's adherence. Only when every judged model fails, and at least two were judged, is it the harness.
One variable at a time. The matrix runs every cell on one model, so the integration level is the only thing that differs. Backends are benchmarked one at a time, alone on the host.
A control for every comparison. A difference between two modes is re-run with the modes alternated before it is attributed to the mode.
Repeat before claiming. Three samples from a clean host, or the number is labelled a single sample. Three benchmark claims were withdrawn for skipping this.
Prove the cause. A fix is tested against a build with the old code. The old build fails and the new one passes.
Evidence is kept. A failed cell keeps its repository, transcript and trace. boxer-eval removes scratch older than three days, and an interrupted run brings down its own sandboxes.
The eval scripts clean up too. A test fails if any benchmark or smoke script allocates a scratch directory without removing it.

What it has caught

Each of these passed the unit tests and was found by an eval tier:

boxer run -c "exit 4" reported exit 1 whenever output went to /dev/null. boxer took /dev/null for a terminal and the runtime refused the TTY request.
One allowlisted hostname that stopped resolving stopped every inside sandbox from starting, because smolvm refuses the whole machine if any allowlisted host fails.
Codex called boxer_status and got a port instead of the URL, because the MCP server it launched used a different state directory from the one that registered the URL.
Inside cells lost their dev server on a host's first round. Caching a harness install as a pack stops the machine, and boxer restarted it without restarting the start services.
A reused worktree path skipped setup, and on docker could see a stale mount.
docker, podman and Apple container published forwarded ports on every interface, so a dev server answered on the Mac's LAN address. Smoke on each backend found it.
On Linux, smolvm ran the image's CMD as the machine's first workload, and when it exited it took the start services with it. The Linux smoke run found it.
Codex ran unsandboxed in linked worktrees, reading project configuration from the main repository.
With a uname shim on PATH, the smolvm launcher's own uname call made boxer call itself and deadlock.

Where the results are

Results has the latest numbers by level, harness and backend. Benchmarks has the costs. The full reports, with every cell's findings, are under docs/ in the repository. Each records the conditions it was made under: boxer and its commit, the backend, the model, the concurrency and the host. Two runs are comparable only when these match.