Results

The latest matrix tier ran two full passes of 55 cells on 2026-09-25. The default pass scored 99.4% of weighted checks. The URLs-mode pass scored 98.7%.

The passes ran three cells at a time. Every cell used zai/glm-5.3-flash, so the integration level is the only variable. Both passes used boxer v1.0.0-48-ged8def6 with uncommitted changes, smolvm 1.16.1 and node:24-bookworm-slim, on a 10-core, 24 GB Apple Silicon Mac. The URLs-mode pass had [urls] on and kept the dev server's address out of the prompt. The default pass is the control. Scorecards are in docs/eval-matrix-report.md and docs/eval-matrix-urls-report.md.

A harness counts as supported at an integration level only if a cell ran and scored there.

By integration level

The inside gap in URLs mode came from a boxer bug unrelated to URLs. Caching a harness install as a pack stops the machine, and boxer did not restart the start services afterwards. The bug showed only on a host's first round of inside cells, and the URLs-mode pass ran first. After the fix, with the pack cache cleared, the inside cells ran cold once in each mode. They passed 10 of 10 and 10 of 10.

Every level has at least two passing harnesses except shell substitution. OpenHands is the only harness whose terminal shell is configurable.

On each backend

The smoke suite ran with no model on the final build on 2026-09-27. Each backend ran alone on its host. No backend failed a test.

• The skips are forks, named packs and the egress log, which only smolvm has (backends).
• The macOS rows ran on an M4 Mac. The Linux rows ran on Ubuntu LTS in a Lima VM on the same Mac, with docker and rootless podman installed natively and smolvm on nested KVM. scripts/linux-smoke.sh ran them.
• The table's macOS smolvm run used smolvm 1.16.1. smolvm also passed 88 of 88 on macOS with smolvm 1.19.2, the current release. Linux used 1.19.2.
• A test checks these rows against the recorded runs in docs/smoke-results.jsonl.

The scripted-model tier (t1) ran 69 cells three times on 2026-09-25. It passed all 68 runnable cells on two of the runs. The third run missed one cell, a herdr pane that inherited the environment of the session running the evaluation. That failure did not recur. The one skipped cell is Multica, which needs a server configured on the host.

When boxer only asks

The adherence tier runs eight harnesses on four live models each, with a prompt that never mentions boxer. The latest report is from 2026-09-25, in docs/eval-adherence.md.

A harness is blamed only when every model fails it. Kimi and Copilot passed on two of four models, so those failures count as model failures. prep is the opt-in host-side step that runs before the sandbox starts. Grok never shows boxer's brief to the model. Its cell fails.

Tool mode

In tool mode the shell tool refuses intercepted commands and the model has to use boxer_run. Through 1.3, the Copilot and Kimi tool-mode cells failed in every recorded run the same way: the model tried the shell first, was refused, and switched. Their brief never reached the model, because neither harness has a hook that injects one at session start, and boxer_run described itself as "for build, test, install, and script commands", which uname is not.

The MCP server now sends the brief as its instructions, and boxer_run's description names what this repository's shell refuses. Measured on 2026-09-30, three runs each:

Known limits

PATH shims fail under a login shell. A login shell rebuilds PATH. On macOS, path_helper puts the system directories first. A harness that runs commands through zsh -lc, as Codex does, reaches the host. boxer cannot put its shims ahead of path_helper, so boxer doctor detects this and names the enforcement to use instead.

Gemini CLI needs its own key. It supports only the Gemini API, which the AI Gateway does not serve, so its cells are skipped.

Conductor is verified by hand. It is a native app, not an Electron app, so it has no debugging port to drive. Its CLI drives cloud workspaces, where boxer has nothing to sandbox.

fx is supported inside the guest only. It has no hooks, its shell cannot be denied headlessly, and it resolves commands past a PATH shim, so boxer has no way to intercept it from outside. Inside the guest it passes.

File watching

A dev server that reloads on save waits for inotify events. This run checked whether an edit made on the host reaches a watcher in the sandbox. Each round made five in-place appends and five write-then-rename saves to files under src/, one second apart. The watcher was chokidar 4 on node:24-alpine. Measured on an M4 Mac on 2026-09-30.

On Apple container and podman the watcher was running, and it saw an edit made inside the sandbox. Only host edits were missing. On smolvm, a plain fs.watch sometimes reported an edit against the directory rather than the file (2 of 5 in-place edits in one round). chokidar still named the file every time.

chokidar is not what Next.js uses, and the smolvm row above does not hold for it. A Next.js 16.3.7 dev server on node:24-bookworm-slim, five edits on the host per cell, each checked by fetching the page for up to 8 seconds:

WATCHPACK_POLLING set on Turbopack changed nothing on Apple container (0 of 5). On Apple container, pollIntervalMs also stopped Turbopack from seeing an edit made inside the sandbox, in both of two tries. Without it, such an edit is seen at once. boxer now sets WATCHPACK_POLLING and CHOKIDAR_USEPOLLING on smolvm, Apple container and podman on a Mac. The Next.js example, created with create-next-app and run with --webpack, then served 10 of 10 host edits on smolvm with nothing set by hand. The smoke suite's file watching cells check the inotify and polling rows on every backend.

Reading a run

Archived runs are in docs/evals/matrix/, newest first, with a JSON line per run. Two runs are comparable only when the model, the concurrency and the host match. Each report records all three.

For a cell below 100%, the scorecard names the claim that failed, and the report keeps the cell's transcript and trace.