Luffy on g-harness-evals: results, costs, and limitations

What the benchmark tests

The benchmark creates a simulated company with Slack conversations, emails, files, and coworkers. Its tasks ask agents to:

• Get approvals before taking the relevant action.
• Use information from earlier conversations and handle corrections.
• Respond to requests in shared channels.
• Keep private information within its permitted scope.
• Combine information from different sources.

The grader checks outcomes such as saved drafts, sent messages, and approval requests. A trial passes only when every required check holds and no consent rule is broken.

These are simulated office tasks. They do not establish performance on coding, every workplace workflow, or live customer workspaces.

How we ran Luffy

Public tasks

20

Repetitions per model

4

Main models

2

Counted trials

160

The counted evaluation ran on 9 November 2026. Before it started, we committed a preregistration covering the configurations, hypotheses, spending limits, exclusion rules, and analysis.

Each trial started in a fresh container with fresh database state. All model calls passed through the benchmark’s metering proxy. All 160 counted trials completed. None was excluded or rerun.

We used Luffy’s existing production code with configuration changes required by the benchmark. The main model stayed fixed within each configuration. Model routing and fallbacks were disabled, along with the context trigger and Docker-based sandbox. Luffy accessed the simulated office through the benchmark’s MCP tools.

Reviewer configuration

Luffy’s response-quality reviewer remained enabled. It uses DeepSeek V4.1 Flash, can request at most one repair, and was declared in advance as a utility model. Every reviewer call was metered and included in the reported cost.

The results therefore measure Luffy with its reviewer. We did not run a controlled reviewer-on versus reviewer-off comparison, so this study cannot tell us how much the reviewer contributed.

Results on GPT-6.1 Sol

Luffy had the highest observed pass rate and the lowest observed model cost per pass in this Sol comparison. Total metered model spend was $9.14, including the reviewer.

Results on Opus 5.5

Luffy again had the highest observed pass rate, but it also had the highest cost per pass in this Opus comparison. Total metered model spend was $29.91, including the reviewer. The cost advantage was specific to Sol.

Compare agents within each model table. Codex and Claude Code were not evaluated against each other on the same model here.

What the numbers include

Pass rate gives each task equal weight

Pass rate is the average of each task’s pass rate, with tasks weighted equally. It is not always total passes divided by total scored trials. That explains why OpenClaw’s Opus figure is 76.3% rather than the direct result of 60/78.

OpenClaw’s smaller denominators come from budget-cap stops excluded by the published baseline’s analysis. We retained the published figures.

Cost per pass includes unsuccessful attempts

Cost per pass is total metered model spend divided by successful scored trials. It includes spend on unsuccessful attempts; the baseline’s spend also includes excluded trials. These are model costs, not the full cost of running the product, such as infrastructure and operations.

Why the pass-rate differences are not conclusive

Luffy’s observed scores were higher. This study does not give us enough evidence to establish that the advantage would hold across a broader set of tasks.

There were only 20 distinct tasks. Four repetitions help measure consistency, but they do not create 80 independent workplace situations per model. Many tasks also produced the same result across systems.

The analysis treated tasks as the unit of comparison and corrected for testing Luffy against several baselines. None of the pass-rate differences remained statistically significant after that correction.

This does not prove the systems are equal. The study cannot confidently separate their performance.

The comparisons were also made across separate runs. The published baseline trials ran on 5 November, while Luffy’s counted run happened on 9 November. Provider behaviour could differ between runs, and the hardware differed. We therefore make no speed claim.

Where Luffy failed

Luffy passed the approval and shared-channel trials listed below. It failed every trial of the two correction tasks shown.

The date-correction and seat-count correction tasks above also scored 0/4 across every published baseline configuration.

When someone changes a date or number, the agent may need to update work already underway. We need to examine the traces to establish why these correction tasks failed.

What this leaves open

All 20 tasks were public, and our team had encountered them during earlier development runs. This limits what we can claim about performance on unseen work.

The evaluation was conducted by the Luffy team. It has not been independently reproduced or certified by the benchmark’s author.

Next, we need to test on unseen tasks under matched run conditions, compare reviewer-on and reviewer-off configurations, and investigate the correction failures.