Matrix

The matrix tier runs the same Next.js workload across every integration level, harness and orchestrator. Each combination is a cell, and each cell gets its own worktree, sandbox and automatic host port. The matrix runs the same two tasks on every configuration. For many tasks on one configuration, see the sdlc tier.

The latest two passes, on 2026-09-25, scored 99.4% of weighted checks by default and 98.7% in URLs mode. Results has the full scores.

make eval-matrix    # 55 cells, three at a time; needs AI_GATEWAY_API_KEY

The two tasks

• add-a-page: write a route that renders a marker and verify it. Every configuration can do this, so a difference between cells is a difference in the level.
• prove-and-install: install a dependency, then record what uname -s says. This catches a level that ran the command on the host.

Scoring

Each cell scores thirteen claims independently, weighted:

If a cell stops early, its unanswered claims count as failures. A provider that refuses a turn makes the cell a skip.

How a cell is verified

The oracle does not accept the agent's platform file alone, because a model can write Linux without running anything. Two checks back it up. The dependency must be in package.json, and boxer's trace must show that level running a command.

The report names which mechanism it found. An orchestrator's launcher gives the harness a clean environment, so boxer's trace variable never reaches it. For orchestrator cells, the oracle checks the worktree the orchestrator created.

Running it

bin/boxer-eval --tier matrix --cell claude/rewrite --parallel 1
bin/boxer-eval --tier matrix --list                        # what would run; runs nothing
BOXER_EVAL_URLS=1 bin/boxer-eval --tier matrix             # every sandbox named, no address in the prompt

BOXER_EVAL_URLS=1 gives every sandbox a portless name. At every outside level, it also stops telling the agent where its dev server is. Each agent has to find its own server from the brief while sibling worktrees run theirs. The oracle still reads the page through the host port boxer allocated, so an agent that used another worktree's server fails the render check. An interrupted run brings down its own sandboxes and removes its scratch.

Every run is archived under docs/evals/matrix/: the full scorecard, an index rebuilt from the data, and a JSON line per run. Each report opens with its conditions: boxer and its commit, smolvm, the model, the image, the concurrency and the host.